The 8th International Conference on Biomedical Engineering and Biotechnology (ICBEB 2019)
October 22-25, 2019, Seoul, Republic of Korea
The China Physiological Signal Challenge 2019:
Challenging QRS Detection and Heart Rate Estimation from Single-Lead ECG Recordings

Invitation Letter (in English)        Invitation Letter (in Chinese)


Welcome to join CPSC 2019 Wechat Group for free discussion and information sharing. Please add the conference wechat and remark with "Please invite me into the Challenge Wechat Group"
     

Notes:
We have noticed that all entries estimate HR from QRS locations, however, HR estimation is an independently awarded event which could be improved by sophisiticated algorithms without QRS information.

[2019-6-21] To solve the confusion caused by the old version of evaluation algorithm while dealing with the boundary problem, we fixed and resubmit all scoring code. Please download the new Sample MATLAB entry and Sample Python entry for your scoring. Meanwhile, some labeling error in training set was fixed and resubmit as Training set!

[2019-5-31] Participants are required to indicate the version of Python or Matlab and the required packages in the Note box during submission.

[2019-5-14] The Sample Python entry is updated for your reference.

[2019-3-26] The last version algorithm of evaluation missed some vital QRS information around the start (0.5s) and end (9.5s) of ECG series, which may result in a lower score. Now we refine it, please download the new Sample MATLAB entry for your scoring.

[2019-3-21] The last 50 data annotations mismatched with recordings because of the overlap in dataset, we have corrected the errors and the training data files in Quick Start have been updated on March 21. Please download the new training set for your code preparation.



The score lists for the first stage has come out as shown below. For those entries which have not been indicated, please check your email to see the error report. Participants are required to indicate the version of Python or Matlab and the required packages in the Note box during submission. Should there be any questions, please feel free to contact chinachallenge_cpsc@icbeb.org.

Score List for Python
Code No.Submitted byInstitution/AffiliationQRSaccHRacc
CPSC0303Yang Liu Harbin Institute of Technology0.82380.8751
CPSC0262Qixun QuShenzhen Digital Life Institute0.81080.8812
CPSC0306Qixun QuShenzhen Digital Life Institute0.78330.8715
CPSC0287Yuan ZhuoChengdu Lanjing Data&Information Technology Co.,LTD0.77690.8664
CPSC0316Shuo ZhangChina Medical University0.72460.8778
CPSC0304Yang QinSoutheast University0.70500.8007
CPSC0296Jie Luo Guangdong University of Technology0.62890.6945
CPSC0265Shuo ZhangChina Medical University0.60670.7995
CPSC0276Tsai-Min ChenMetformin-1210.58470.7396
CPSC0310Jin QiUniversity of Electronic Science and Technology of China,
Chengdu Lanjing Data&Information Technology Co.,LTD
0.47390.6301
CPSC0289Chuang HanZhengzhou University0.40830.6582
CPSC0298Wei HanChengdu Lanjing Data&Information Technology Co.,LTD0.36990.5665


Score List for Matlab
Code No.Submitted byInstitution/AffiliationQRSaccHRacc
CPSC0315Lishen QiuSoochow University0.78490.8871
CPSC0305Juan FuQilu University of Technology0.54940.8032
CPSC0275Lishen QiuSoochow University0.54790.7078
CPSC0270Xin ZhangSoutheast University0.45600.6238
CPSC0288Wei LiLudong Univeristy0.43230.6607
CPSC0311Chuanbin GeBeijing Institute of Technology0.38590.6030
CPSC0309Xiuzhu YangBeijing University of Posts and Telecommunications0.34090.5198
CPSC0260Chunli WangChengdu Spaceon Electronics CO.,LTD.0.32070.5183
CPSC0259Jianping LinTsinghua University0.31070.5107


Frequently Asked Questions (FAQs)

Q1: How can I register to attend this challenge?
R1: Please click http://www.cpscsub.com/ to complete the registration.

Q2: How can we register team information to join the challenge?
R2: We suggest that the team leader, or one of the team representatives registers an account on behalf of the team in the system.

Q3: The sample entry, ‘CPSC2019_challenge.m’, is unable to be downloaded.
R3: Right click, save link as, and then download is finished.

Q4: How should I submit the challenge entries with the provided sample entry ‘CPSC2019_challenge’?
R4: You can add the ECG signal processing codes in the ‘CPSC2019_challenge.m', and submit the updated version of 'CPSC2019_challenge.m'. At this moment, please focus on the methods you need to develop, and submit the ‘CPSC2019_challenge.m’ with developed codes. Test data and training data are exactly in the same format.

Q5: How should I name the main function?
R5: Please name it CPSC2019_challenge.

Q6: If we use the python environment, can the test system support commonly used packages, such as keras, wfdb, numpy?
R6: Yes, It can.

Introduction

The China Physiological Signal Challenge 2019 (CPSC 2019) aims to encourage the development of algorithms for challenging QRS detection and heart rate (HR) estimation from short-term single-lead ECG recordings usually with low signal quality and/or abnormal rhythm waveforms.

ECG signal provides an important role in non-invasively monitoring and clinical diagnosis for cardiovascular disease (CVD). Detection of QRS complex is an essential step for ECG signal processing, and can benefit the following HR calculation and abnormal situation analysis. Although detection methods of QRS complex have been severely tracked throughout the last several decades, accurate QRS location and HR estimation are still challenging in noisy signal episode or abnormal rhythm waveforms, especially when the ECG recordings are from the wearable dynamic ECG acquisition. It is true that many of the developed QRS detection algorithms can achieve high accuracy (over 99% in sensitivity and positive predictivity) when tested over the standard ECG databases such as MIT-BIH Arrhythmia Database or AHA Database [1]. However, these algorithms may not be able to perform well when used in the daily life environment that will cause severe noises and significantly reduce the signal quality. A recent study confirmed that none of the common QRS algorithms can obtain 80% detection accuracy when tested in a common dynamic noisy ECG database [2]. Thus, in this challenge, we provide a new ECG database containing noisy ECG episodes and/or signals with different arrhythmia patterns, encouraging participants to develop more efficient and robust algorithms QRS detection and HR estimation. In addition, it is worth to note that, although HR can be calculated from the detection results of QRS complexes, HR can be still estimated without QRS detection step [3,4].

Challenge Data

Training data consists of 2,000 single-lead ECG recordings collected from patients with cardiovascular disease (CVD), each of the recording last for 10 s. Test set contains similar ECG recordings of same lengths, which is unavailable to public and will remain private for the purpose of scoring for the duration of the Challenge and for some period afterwards. ECG recordings were obtained from multiple sources using a variety of instrumentation, although in all cases they are presented as 500 Hz sample rate here. All recordings were provided in MATLAB format (each including two .mat file: one is ECG data and another one is the corresponding QRS annotation file). Pan &Tompkins (P&T) algorithm [5,6] is also provided as benchmark or comparable algorithm.

Although QRS detection and HR estimation are widely studied by lots of researchers for many years, accurate detection is still really challenging in this Challenge due to the QRS amplitude variation, QRS morphological variation, and occurrence of intense variability in the intervals between beats, different arrhythmias, as well as noises. Figure 1 shows the examples of ECG waveforms in training data.



Figure 1. Examples of the represented ECG waveforms. Red circles denote the reference QRS locations and black ones denote the detected results by the P&T algorithm. The challenges for accurate QRS locations are from: A) intense variability, B) premature ventricular contraction (PVC) and C) ventricular tachycardia.

Challenge Events and Scoring

CPSC 2019 is comprised of two events related to scoring: QRS location and HR estimation. Only test set will be used for event scoring. QRS annotations in the training and test sets are labeled and approved by cardiologists and trained volunteers. These reference annotations have been processed to derive the reference RR interval time series (the intervals between successive QRS annotations). HR is further derived from the RR interval with a 4 s time window ranging from 5.5 s to 9.5 s in each of 10-s ECG segment.

Event 1: QRS detection

In this event, the goal is to produce a set of QRS annotations that can matches the reference QRS annotations. For each reference QRS annotation, a matched QRS annotation should lie in 75 ms duration centered by the reference QRS annotation [9]. Noted that the reference QRS annotations appear in the first and last half seconds are omitted. As shows in Figure 1, detected QRS must be within 75 ms from the reference ones. For each 10-s ECG segment, the scoring rules are:
• complete matching scores one point
• a false positive (FP) detection scores 0.7 points
• a false negative (FN) detection scores 0.3 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection here
• other situations score 0

            

The final score for Event 1 (QRSacc) can be calculated as:

                 

Event 2: HR estimation

In this event, the goal is to produce a real-time estimate for HR for a 4-s ECG episode, i.e., ECG signals from 5.5 s to 9.5 s. Figure 2 demonstrates the evaluation method for Event 2, i.e., how HRscore for each 10-s ECG segment can be obtained.

Figure 2. Demonstration of the evaluation method for Event 2. In this situation, reference HR can be calculated from the reference QRS annotations between 5.5 s and 9.5 s ECG episode (HRref = 96), and estimated HR can be calculated from the detected QRS annotations between the same 4-s ECG episode (HRtest = 76).
The detailed scoring rules for HR estimation in each 10-s ECG segment are summarized as:

                  

The final score for Event 2 (HRacc) can be calculated as:

               

where HRtest and HRref represent heart rate calculated by competitor and reference, respectively. HR estimation is usually derived from the QRS detection but we encourage participant to develop robust HR estimation without QRS location.

Quick Start

1. Download the Training set the Sample MATLAB entry and Sample Python entry (updated on June 21, 2019)
2. Create a free account and join the CPSC Challenge 2019 project.
3. Develop your entry by editing the existing files:
     •  Modify the sample entry source code file CPSC2019_challenge.m with your changes and improvements.
     •  Modify the AUTHORS.txt file to include the names of all the team members.
     •  Run your modified source code file on all the records in the training set by executing the script CPSC2019_ccore.m. This will also build a new version of
        entry.zip.
     •  Optional: Include a file named DRYRUN in the top directory of your entry (where the AUTHORS.txt file is located) if you do not wish your entry to be
        scored and counted against your limit. This is useful in cases where you wish to make sure that the changes made do not result in any error.
4. Submit your modified entry.zip for scoring through the CPSC Challenge 2019 project. The contents of entry.zip must be laid out exactly as in the sample entry. Improperly-formatted entries will not be scored.

Participating in the challenge

As a starting point, we have provided example entries (Sample MATLAB entry/ Sample Python entry) implemented using Matlab or Python, which provides the most widely used QRS peaks annotation algorithm, pan_tompkin.m, as benchmark or comparable algorithm.
To participate in the challenge, you will need to submit a software written by MATLAB or Python that is able to run on the test set and output the final scoring result without user interaction in our test environment. One sample entry CPSC2019_challenge.m is available to help you get started. And a scoring function CPSC2019_score.m is provided for competitors testing algorithms and build a new version of entry.zip. The team must submit the first challenge open-source entry before June 1, 2019. The deadline for submitting the official challenge entries is October 1, 2019.
• Please use the following link to upload your codes, http://www.cpscsub.com/ .

Awards and Rules

The winner will be selected on the basis of the obtained final QRSacc and HRacc on the hidden test data. The first three for each Event challenging will receive certificates and generous bonuses:
• First prize: Certificate plus bonus of RMB 15,000
• Second prize: Certificate plus bonus of RMB 10,000
• Third prize: Certificate plus bonus of RMB 5,000

We welcome all the individual or research group around the world to attend the challenge. To be eligible for the award, please do the following:
(Please be noted that paper submission is not mandatory and will not affect the challenge score.)
(1) Register the team before May 1, 2019.
(2) Submit at least one open-source entry that can be scored before June 1, 2019.
(3) Attend ICBEB 2019 (October 22-25, 2019) in Seoul, Republic of Korea and present your work there.

Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after ICBEB 2019 has taken place, so the competitors are able to discuss the results in a single forum.

Important Dates

January 25, 2019 -- Challenge open
May 1, 2019 -- Deadline for team registration to join the challenge
June 1, 2019 -- Deadline for submitting the first challenge open-source entry
October 1, 2019 -- Deadline for submitting the challenge entries
October 22-25, 2019 – Announcement of the winners of CPSC 2019 in ICBEB 2019

Any questions or problems about the Challenge, please feel free to contact chinachallenge_cpsc@icbeb.org.

Publication – Call for Chapters

The book with Springer Nature entitled Feature Engineering and Computational Intelligence in ECG Monitoring calls for chapters.

The purpose of this book is to summarize the feature engineering and computational intelligence solutions for ECG monitoring, with an emphasis on how these methods can be efficiently used on the emerging need and challenge -- dynamic, continuous & long-term individual ECG monitoring and real-time feedback, aiming to provide a “snapshot” of the state of current research at the interface between physiological signal analysis and machine learning. It could help clarify some dilemmas and encourage further investigations in this field, to explore rational applications of feature engineering and computational intelligence in clinical practices for ECG monitoring.

Original, high quality contributions that are not yet published or that are not currently under review by other journals or peer-reviewed conferences are welcomed, with special emphasis on, but not limited to, a number of correlative chapter topics.

Click Book Proposal for more details.

Challenge Chair:
Chengyu Liu, Southeast University, China

Challenge Committee:
Hongxiang Gao, Southeast University, China
Xingyao Wang, Southeast University, China
Feifei Liu, Southeast University, China
Lina Zhao, Southeast University & Shandong University, China
Xiaoling Wu, Nanjing Medical University, China

International Advisory Chair:
Gari D. Clifford, Emory University & Georgia Institute of Technology, USA

International Advisory Co-chairs:
Aiguo Song, Southeast University, China
Jianqing Li, Nanjing Medical University, China
Zuhong Lu, Southeast University, China
Ye Li, SIAT of Chinese Academy of Sciences, China
Yingjia Yao, Lenovo Group, China

International Advisory Committee:
Eddie Ng Yin Kwee, Nanyang Technological University, Singapore
Amit Shah, Emory University, USA
Yi Peng, Peking Union Medical College, China
Shoushui Wei, Shandong University, China
Zhengtao Cao, Aviation Medicine Institute, China
Hongxing Liu, Nanjing University, China
Fengfeng Zhou, Jilin University, China
Alistair Johnson, MIT, USA

Hosted by:
School of Instrument Science and Engineering, Southeast University, China
The State Key Laboratory of Bioelectronics, Southeast University, China
School of Biomedical Engineering and Information, Nanjing Medical University, China

Supported by:
Health Engineering Branch of Chinese Society of Biomedical Engineering
ICBEB Organizing Committee

Awards sponsored by:
Lenovo Group

Reference
1.   G.B. Moody, R.G. Mark, The impact of the MIT-BIH arrhythmia database, IEEE Engineering in Medicine & Biology Magazine the Quarterly Magazine of
      the Engineering in Medicine & Biology Society, 20 (2001) 45-50.
2.   Liu, F.F.; Wei, S.S.; Li, Y.B.; Jiang, X.E.; Zhang, Z.M.; Liu, C.Y., Performance analysis of ten common qrs detectors on different ecg application cases.
      Journal of Healthcare Engineering 2018, 2018, ID 9050812.
3.   J.J. Gieraltowski, K. Ciuchcinski, I. Grzegorczyk, K. Kosna, Heart rate variability discovery: Algorithm for detection of heart rate from noisy, multimodal
      recordings, Computing in Cardiology, 2014, pp. 253-256.
4.   J. Gieraltowski, K. Ciuchcinski, I. Grzegorczyk, K. Kosna, M. Solinski, P.Podziemski, RS slope detection algorithm for extraction of heart rate from noisy,
      multimodal recordings, Physiological Measurement, 36 (2015) 1743-1761.
5.   P.S. Hamilton, W.J. Tompkins, Quantitative investigation of QRS detection rules using the MIT/BIH arrhythmia database, Biomedical Engineering, IEEE
      Transactions on, (1986) 1157-1165.
6.   J. Pan, W.J. Tompkins, A real-time QRS detection algorithm, Biomedical Engineering, IEEE Transactions on, (1985) 230-236.
7.   ANSI-AAMI (1998). Testing and reporting performance results of cardiac rhythm and st segment measurement algorithms, ANSI-AAMI:EC57.


                                                  


The 8th International Conference on Biomedical Engineering and Biotechnology
Conference Secretaries: Ms. Linda Li, Ms. Annie Zhu, Ms. Cassie Cheng
Email: icbeb@icbeb.org / icbeb@academicconf.com
Tel: +86-27-87051286  Cell phone/whatsApp: +86-13018020541
Copyright © ICBEB 2019! All rights reserved.