Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria

Bo Remenyi, Jonathan Carapetis, John W. Stirling, Beatrice Ferreira, Krishnan Kumar, John Lawrenson, Eloi Marijon, Mariana Mirabel, A. O. Mocumbi, Cleonice Mota, John Paar, Anita Saxena, Janet Scheel, Satu Viali, I. B. Vijayalakshmi, Gavin R. Wheaton, Liesl Zuhlke, Karishma Sidhu, Eliazar Dimalapang, Thomas L. Gentles & 1 others Nigel J. Wilson

    Research output: Contribution to journalArticleResearchpeer-review

    Abstract

    Objective: Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria.

    Methods: A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen's and Fleiss' free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria.

    Results: Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ.

    Conclusions: The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.

    Original languageEnglish
    Article numbere011233
    Pages (from-to)1-8
    Number of pages8
    JournalHeart Asia
    Volume11
    Issue number2
    DOIs
    Publication statusPublished - 24 Jun 2019

    Fingerprint

    Rheumatic Heart Disease
    Aortic Valve Insufficiency
    Mitral Valve Insufficiency
    Sample Size
    Disease Progression

    Cite this

    Remenyi, Bo ; Carapetis, Jonathan ; Stirling, John W. ; Ferreira, Beatrice ; Kumar, Krishnan ; Lawrenson, John ; Marijon, Eloi ; Mirabel, Mariana ; Mocumbi, A. O. ; Mota, Cleonice ; Paar, John ; Saxena, Anita ; Scheel, Janet ; Viali, Satu ; Vijayalakshmi, I. B. ; Wheaton, Gavin R. ; Zuhlke, Liesl ; Sidhu, Karishma ; Dimalapang, Eliazar ; Gentles, Thomas L. ; Wilson, Nigel J. / Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria. In: Heart Asia. 2019 ; Vol. 11, No. 2. pp. 1-8.
    @article{8ce05acf5a674c8bad3a719e6244593c,
    title = "Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria",
    abstract = "Objective: Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria. Methods: A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen's and Fleiss' free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria. Results: Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ. Conclusions: The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.",
    keywords = "aortic valve disease, mitral regurgitation, paediatric echocardiography, rheumatic fever",
    author = "Bo Remenyi and Jonathan Carapetis and Stirling, {John W.} and Beatrice Ferreira and Krishnan Kumar and John Lawrenson and Eloi Marijon and Mariana Mirabel and Mocumbi, {A. O.} and Cleonice Mota and John Paar and Anita Saxena and Janet Scheel and Satu Viali and Vijayalakshmi, {I. B.} and Wheaton, {Gavin R.} and Liesl Zuhlke and Karishma Sidhu and Eliazar Dimalapang and Gentles, {Thomas L.} and Wilson, {Nigel J.}",
    year = "2019",
    month = "6",
    day = "24",
    doi = "10.1136/heartasia-2019-011233",
    language = "English",
    volume = "11",
    pages = "1--8",
    journal = "Heart Asia",
    issn = "1759-1104",
    publisher = "BMJ Publishing Group",
    number = "2",

    }

    Remenyi, B, Carapetis, J, Stirling, JW, Ferreira, B, Kumar, K, Lawrenson, J, Marijon, E, Mirabel, M, Mocumbi, AO, Mota, C, Paar, J, Saxena, A, Scheel, J, Viali, S, Vijayalakshmi, IB, Wheaton, GR, Zuhlke, L, Sidhu, K, Dimalapang, E, Gentles, TL & Wilson, NJ 2019, 'Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria', Heart Asia, vol. 11, no. 2, e011233, pp. 1-8. https://doi.org/10.1136/heartasia-2019-011233

    Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria. / Remenyi, Bo; Carapetis, Jonathan; Stirling, John W.; Ferreira, Beatrice; Kumar, Krishnan; Lawrenson, John; Marijon, Eloi; Mirabel, Mariana; Mocumbi, A. O.; Mota, Cleonice; Paar, John; Saxena, Anita; Scheel, Janet; Viali, Satu; Vijayalakshmi, I. B.; Wheaton, Gavin R.; Zuhlke, Liesl; Sidhu, Karishma; Dimalapang, Eliazar; Gentles, Thomas L.; Wilson, Nigel J.

    In: Heart Asia, Vol. 11, No. 2, e011233, 24.06.2019, p. 1-8.

    Research output: Contribution to journalArticleResearchpeer-review

    TY - JOUR

    T1 - Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria

    AU - Remenyi, Bo

    AU - Carapetis, Jonathan

    AU - Stirling, John W.

    AU - Ferreira, Beatrice

    AU - Kumar, Krishnan

    AU - Lawrenson, John

    AU - Marijon, Eloi

    AU - Mirabel, Mariana

    AU - Mocumbi, A. O.

    AU - Mota, Cleonice

    AU - Paar, John

    AU - Saxena, Anita

    AU - Scheel, Janet

    AU - Viali, Satu

    AU - Vijayalakshmi, I. B.

    AU - Wheaton, Gavin R.

    AU - Zuhlke, Liesl

    AU - Sidhu, Karishma

    AU - Dimalapang, Eliazar

    AU - Gentles, Thomas L.

    AU - Wilson, Nigel J.

    PY - 2019/6/24

    Y1 - 2019/6/24

    N2 - Objective: Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria. Methods: A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen's and Fleiss' free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria. Results: Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ. Conclusions: The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.

    AB - Objective: Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria. Methods: A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen's and Fleiss' free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria. Results: Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ. Conclusions: The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.

    KW - aortic valve disease

    KW - mitral regurgitation

    KW - paediatric echocardiography

    KW - rheumatic fever

    UR - http://www.scopus.com/inward/record.url?scp=85068190993&partnerID=8YFLogxK

    U2 - 10.1136/heartasia-2019-011233

    DO - 10.1136/heartasia-2019-011233

    M3 - Article

    VL - 11

    SP - 1

    EP - 8

    JO - Heart Asia

    JF - Heart Asia

    SN - 1759-1104

    IS - 2

    M1 - e011233

    ER -