The 9th International Conference on Biomedical Engineering and Biotechnology (ICBEB 2020)
November 15-18, 2020, Suzhou, China

Searching for Premature Ventricular Contraction and Supraventricular Premature Beat from Long-term ECGs: The 3rd China Physiological Signal Challenge 2020



If you use the Challenge data for paper publication, please cite this paper for Challenge data description:
H. X. Gao, C. Y. Liu, X. Y. Wang, L. N. Zhao, Q. Shen, E. Y. K. Ng, and J. Q. Li. An Open-Access ECG Database for Algorithm Evaluation of QRS Detection and Heart Rate Estimation. Journal of Medical Imaging and Health Informatics, 2019, 9(9): 1853–1858.

Invitation Letter (in English)        Invitation Letter (in Chinese)


Welcome to join CPSC 2020 Wechat Group for free discussion and information sharing. Please add the conference wechat and remark with "Please invite me into the Challenge Wechat Group"
     

Frequently Asked Questions (FAQs)
Q1: How can I register to attend this challenge?
R1: Please click http://www.cpscsub.com/ to complete the registration.

Q2: How can we register team information to join the challenge?
R2: We suggest that the team leader, or one of the team representatives registers an account on behalf of the team in the system. Please DO NOT register multiple accounts from one team.

Q3: How should I submit the challenge entries with the provided sample entry ‘CPSC2020_challenge’?
R3: You can add the signal processing codes in the ‘CPSC2020_challenge.m', and submit the updated version of 'CPSC2020_challenge.m'. At this moment, please focus on the methods you need to develop, and submit the ‘CPSC2020_challenge.m’ with developed codes. Test data and training data are exactly in the same format.

Q4: How should I name the main function?
R4: Please name it CPSC2020_challenge.

Q5: If we use the python environment, can the test system support commonly used packages, such as keras, wfdb, numpy?
R5: Yes, It can.

Q6: How can I contact the organizing committee?
R6: Please join the Wechat Group or email to chinachallenge_cpsc@icbeb.org.

Introduction
Abnormality of cardiac conduction system can induce arrhythmia. Abnormal heart rhythm can lead to other cardiac diseases and complications, and can be life-threatening [1]. There are various types of arrhythmias and each type is associated with a pattern, and as such, it is possible to be identified. Arrhythmias can be classified into two major categories. The first category consists of arrhythmias formed by a single irregular heartbeat in electrocardiogram (ECG), herein called morphological arrhythmia, while another category consists of arrhythmias formed by a set of irregular heartbeats in ECG, herein called rhythmic arrhythmias [2]. Dynamic electrocardiogram (DCG), like ECG Holter, provides an important way to monitor the incidences of arrhythmias in daily life, facilitating the doctors to check a total number and distribution of arrhythmias in a long time and thus to provide the required therapy to prevent further problems.
The 3rd China Physiological Signal Challenge 2020 (CPSC 2020) aims to encourage the development of algorithms for searching for premature ventricular contraction (PVC) and supraventricular premature beat (SPB) from 24-hour dynamic single-lead ECG recordings usually with low signal quality and/or abnormal rhythm waveforms. Similar the previous works and efforts of the CPSC 2018 [3] and CPSC 2019 [4], accurate locating of abnormal heartbeats is another critical issue put forward here for further discussion.
ECG signal provides an important role in non-invasively monitoring and clinical diagnosis for cardiovascular disease (CVD). Arrhythmia detection is one of the ultimate goals of routine ECG monitoring, and PVC and SPB are the two most common arrhythmias. Increase in these beats may be a precursor to stroke or sudden cardiac death [5]. Although their detection methods have been severely tracked throughout the last several decades, accurate and robust detections are still challenging in noisy or low-signal quality environment, especially for daily monitored ECG waveforms. It is true that many of the developed PVC and SPB detection algorithms can achieve high accuracy (over 96% in sensitivity and positive predictivity) when tested over the standard ECG databases such as the MIT-BIH Arrhythmia Database or AHA Database [6]. However, these algorithms may fail when used in the noisy environment. Especially, even the basic QRS detection can be invalid in the low signal quality ECG analysis [7]. A recent study confirmed that none of the common QRS detection algorithms can obtain 80% detection accuracy when tested in a dynamic noisy ECG database. In this year’s challenge, we provide a new ECG database containing long-term noisy ECG recordings from clinical arrhythmia patients, to encourage the participants to develop more efficient and robust algorithms for PVC and SPB detection.

Challenge Data
Training data consists of 10 single-lead ECG recordings collected from arrhythmia patients, each of the recording last for about 24 hours (shown in Table 1). Table 1 also indicates the patient if he/she is an atrial fibrillation (AF) patient. Test set contains similar ECG recordings, which is unavailable to public and will remain private for the purpose of scoring for the duration of Challenge and for some period afterwards. All data were collected by a unified wearable ECG device with a sampling frequency of 400 Hz, and provided in MATLAB format (each including three *.mat file: one is ECG data and another two are the corresponding PVC and SPB annotation files, respectively).

Table 1: Detailed information of training data.
RecordingsAF patient ?Length (h)# N beats# V beats# S beats# Total beats
A01No25.89109,062024109,086
A02Yes22.8398,9364,5540103,490
A03Yes24.70137,2493820137,631
A04No24.5177,81219,0243,466100,302
A05No23.5794,61412594,640
A06No24.5977,6210677,627
A07No23.1173,32515,1503,48191,956
A08Yes25.46115,5182,7930118,311
A09No25.8488,22921,46289,693
A10No23.6472,8211699,07182,061

Although PVC and SPB detection algorithms are widely studied for many years, accurate detection is still challenging in this Challenge due to the amplitude variation, morphological variation, as well as noises in ECG data. Figure 1 shows examples of ECG waveforms in training data.


Figure 1. Examples of demonstrated ECG waveforms. Black circles denote the reference locations of SPB and red ones denote the reference locations of PVC. The challenges for accurate PVC and SPB detection are from: A) amplitude variability and multi-source PVC; B) noise and other abnormal interference (such as AF) and C) electrode sliding interference.

Challenge Events and Scoring
CPSC 2020 is comprised of two events related to scoring: PVC detection and SPB detection. PVC and SPB annotations in the training and test sets are labeled and initially confirmed by cardiologists and trained volunteers. Score is calculated according to the following rules.
Event 1: PVC detection
In this event, the goal is to generate a set of PVC annotations for each recording that can matches the reference PVC annotations. For each reference PVC annotation, a matched PVC annotation should lie in 150 ms duration centered by the reference PVC annotation [8]. Noted that the reference PVC annotations appear in the first and last 0.2 seconds are ignored. Detected PVC should be within 150 ms from the reference ones. The scoring rules are:
    •   a false positive (FP) detection deduct 1 point.
    •   a false negative (FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
        The final score for Event 1 (PVCerr) is the sum of all deducted points.
Event 2: SPB detection
In this event, the goal is to generate a set of SPB annotations for each recording that can matches the reference SPB annotations. For each reference SPB annotation, a matched SPB annotation should lie in 150 ms duration centered by the reference SPB annotation [8]. Noted that the reference SPB annotations appear in the first and last 0.2 seconds are also ignored. Detected SPB should be within 150 ms from the reference ones. The scoring rules are:
    •   a false positive (FP) detection deduct 1 point.
    •   a false negative (FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
        The final score for Event 2 (SPBerr) is the sum of all deducted points.

Quick Start
1. Download the training set and the sample entry (Sample MATLAB entry/ Sample Python entry ).
2. Create a free account and join the CPSC Challenge 2020 project.
3. Develop your entry by editing the existing files:
    •   You are allowed modify the sample entry source code file CPSC2020_challenge.m with your changes and improvements. But, never alter any word in CPSC2020_score.m to avoid a wrong result.
    •   Modify the AUTHORS.txt file to include the names of all the team members.
    •   Run your modified source code file on all the records in the training set by executing the script CPSC2020_score.m. This will also build a new version of entry.zip.
    •   Optional: Include a file named DRYRUN in the top directory of your entry (where the AUTHORS.txt file is located) if you do not wish your entry to be scored and counted against your limit. This is useful in cases where you wish to make sure that the changes made do not result in any error.
4. Submit your modified entry.zip for scoring through the CPSC Challenge 2020 project. The contents of entry.zip must be laid out exactly as in the sample entry. Improperly-formatted entries will not be scored.
5. Please note: we cannot guarantee the 100% accuracy for the reference PVC/SPB annotations. If you find that the reference is questioned (error), please give us a feedback .txt file using the following format to us:
File name: "data label name - event type (S / V) - sampling point position - expected result" (example: A07-S-76214-N.txt). Please keep the ‘.txt’ content empty.
We will further verify this with the cardiologists, and update the reference annotations in the website periodically.

Participating in the challenge
To participate in the challenge, you need to submit a software written by MATLAB or Python that is able to run on the test set and output the final scoring result without user interaction in our test environment. One sample entry CPSC2020_challenge.m is available to help you get started. And a scoring function CPSC2020_score.m is provided for competitors testing algorithms and build a new version of entry.zip. The team must submit the first challenge open-source entry before July 1, 2020. The deadline for submitting the official challenge entries is October 25, 2020.
    •   Please use the following link to upload your codes, http://www.cpscsub.com/ .

Awards and Rules
The winner will be selected on the basis of the obtained final PVCerr and SPBerr on the hidden test data. The first three for each Event challenging will receive certificates and generous bonuses:
    •    First prize: Certificate plus bonus of RMB 15,000
    •   Second prize: Certificate plus bonus of RMB 10,000
    •   Third prize: Certificate plus bonus of RMB 5,000
We welcome all the individual or research group around the world to attend the challenge. To be eligible for the open-source award, please do the following (see table 2):
    (1) Register the team before June 1, 2020.
    (2) Submit at least one open-source entry that can be scored before July 1, 2020.
    (3) Attend ICBEB 2020 (November 15-18, 2020) in Suzhou, China and present your work there.
All deadlines occur at 11:59pm GMT+8 (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!

Table 2: Challenge deadlines.
StartEntry limitEnd
Registration1 FebruaryAt least once30 June
First Stage1 July1031 September
Second Stage1 October325 October
Final Stage26 OctoberFinal entry1 November

In the interest of fairness to all participants, any entries received later than deadline of that stage will not be accepted or scored. Code submitted more than the limit number of times will not be scored.
Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after ICBEB 2020 has taken place, so the competitors are able to discuss the results in a single forum.

Important Dates
15 February, 2020 -- Challenge open
June 1, 2020 -- Deadline for team registration to join the challenge
July 1, 2020 -- Deadline for submitting the first challenge open-source entry
October 25, 2020 -- Deadline for submitting the challenge entries
November 1, 2020 -- Confirm for submitting the best entry for final scoring
November 15-18, 2020 – Announcement of the winners of CPSC 2020 in ICBEB 2020

Any questions or problems about the Challenge, please feel free to contact chinachallenge_cpsc@icbeb.org

Challenge Chair:
Prof. Chengyu Liu, Southeast University, China
Challenge Committee:
Dr. Zhipeng Cai, Southeast University, China
Dr. Hongxiang Gao, Southeast University, China
Dr. Xingyao Wang, Southeast University, China
Dr. Qin Shen, Nanjing Medical University, China
Dr. Xiangwei Ding, Nanjing Medical University, China
International Advisory Chair:
Prof. Gari D. Clifford, Emory University & Georgia Institute of Technology, USA
International Advisory Co-chairs:
Prof. Aiguo Song, Southeast University, China
Prof. Jianqing Li, Nanjing Medical University, China
Prof. Zuhong Lu, Southeast University, China
Prof. Yi Peng, Peking Union Medical College, China
Mr. Yingjia Yao, Lenovo Group, China
International Advisory Committee:
Prof. Eddie Ng Yin Kwee, Nanyang Technological University, Singapore
Prof. Kang-Ping Lin, Chung Yuan Christian University, Chinese Taipei
Prof. Feng Wang, Southeast University, China
Prof. Zhengtao Cao, Air Force Medical Center, China
Prof. Xianzheng Sha, China Medical University, China
Prof. Xingming Guo, Chongqing University, China
Prof. Shoushui Wei, Shandong University, China
Prof. Zhi Tao, Suzhou University, China
Dr. Alistair Johnson, MIT, USA

Hosted by:
School of Instrument Science and Engineering, Southeast University, China
The State Key Laboratory of Bioelectronics, Southeast University, China
School of Biomedical Engineering and Information, Nanjing Medical University, China

Supported by:
ICBEB Organizing Committee
Health Engineering Committee of Chinese Society of Biomedical Engineering
Biomedical Sensor Committee of Chinese society of Biomedical Engineering
Youth Working Committee of Chinese Society of Biomedical Engineering
Jiangsu Instrument and Control Society

Awards sponsored by:
Lenovo Group

Reference
[1] S. L. Oh, E. Y. Ng, R. San Tan, and U. R. Acharya, "Automated diagnosis of arrhythmia using combination of CNN and LSTM techniques with variable length heart beats," Computers in biology and medicine, vol. 102, pp. 278-287, 2018.
[2] E. J. D. S. Luz, W. R. Schwartz, G. Cámara-Chávez, and D. Menotti, "ECG-based heartbeat classification for arrhythmia detection: A survey," Computer methods and programs in biomedicine, vol. 127, pp. 144-164, 2016.
[3] F. Liu, C. Liu, L. Zhao, X. Zhang, X. Wu, X. Xu, Y. Liu, C. Ma, S. Wei, Z. He, J. Li, and E. Y. K. Ng, "An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection," Journal of Medical Imaging and Health Informatics, vol. 8, pp. 1368-1373, 2018.
[4] H. Gao, C. Liu, X. Wang, L. Zhao, Q. Shen, E. Y. K. Ng, and J. Li, "An Open-Access ECG Database for Algorithm Evaluation of QRS Detection and Heart Rate Estimation," Journal of Medical Imaging and Health Informatics, vol. 9, pp. 1853-1858, 2019.
[5] J. Oster, J. Behar, O. Sayadi, S. Nemati, A. E. Johnson, and G. D. Clifford, "Semisupervised ECG ventricular beat classification with novelty detection based on switching Kalman filters," IEEE Transactions on Biomedical Engineering, vol. 62, pp. 2125-2134, 2015.
[6] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. Peng, and H. E. Stanley, "PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals," Circulation, vol. 101, pp. e215-e220, 2000.
[7] F. Liu, C. Liu, X. Jiang, Z. Zhang, Y. Zhang, J. Li, and S. Wei, "Performance analysis of ten common QRS detectors on different ECG application cases," Journal of Healthcare Engineering, vol. 2018, pp. 9050812(1)-9050812(8), 2018.
[8] ANSI/AAMI EC57, "1998 / (R) 2008-Testing and reporting performance results of cardiac rhythm and ST segment measurement algorithms", Arlington, VA, USA, 2008.