FRAGRANCE REPEAT-AGREEMENT ANALYSIS

Original file: C:\Users\reduc\Documents\famous_fragrance50_data_raw.csv
Corrected file: C:\Users\reduc\Documents\famous_fragrance50_data_raw_corrected.csv
Generated: 2026-09-11 22:43:51 EDT

KEY OUTPUTS
figures/Main_text_confusion_matrices.pdf: vector, two columns x three rows, US-letter portrait.
figures/Main_text_confusion_matrices.png: same figure at the configured PNG resolution.
figures/Main_text_confusion_matrices.tiff: LZW-compressed 600-dpi figure when supported.
figures/Supplement_all_available.pdf: every matrix, all available pairs in each file; six matrix columns by default.
figures/Supplement_matched_complete.pdf: all pairs using matched complete cases, when enabled.
tables/pair_map.csv: exact question names, column indices, occurrence numbers, and pair IDs.
tables/agreement_statistics.csv: all coefficients, ICC confidence limits, sample sizes, exact agreement, and diagnostics.
tables/original_vs_corrected.csv: side-by-side statistics and corrected-minus-original differences for both sample definitions.
tables/main_text_selection.csv: selected rows, statistics, and explicit selection rules.
tables/confusion_matrix_cells.csv: all cell counts and whole-matrix proportions.
diagnostics/input_quality.csv: missingness and changes in response availability.
diagnostics/sample_membership.csv: which fragrances enter each available/matched comparison.
diagnostics/estimation_notes.csv: warnings, constant variables, and near-boundary estimates.

MISSINGNESS
Blanks are missing, not zero, not an additional category, and not an implicit response of 2.
Available means pairwise complete within one file, not complete across all items.
Matched means both occurrences are observed in both files for that particular pair.
The corrected data change response availability, so all-available differences combine rating changes and sample-composition changes.
Matched comparisons isolate changes on a fixed subset but do not establish unbiased generalization beyond complete cases.

STATISTICAL DEFINITIONS
Polychoric r: polycor::polychor, method = two_step, maxcor = 0.9999, no pseudocount/continuity correction.
Polychoric correlation treats 0<1<3<4 as ordered categories and assumes an underlying bivariate normal model; sparse margins can make the estimate unstable.
Unused marginal categories are dropped only for polychoric estimation. All four categories remain visible in every matrix.
ICC(A,1): irr::icc(model='twoway', type='agreement', unit='single'), checked against its ANOVA formula.
ICC = (MS_subject-MS_error)/(MS_subject+(k-1)*MS_error+k*(MS_occurrence-MS_error)/n), with k=2.
ICC retains the stated 0,1,3,4 metric, so 1-to-3 disagreement is larger than 0-to-1 disagreement. Its interpretation assumes those numeric distances are meaningful.
ICC confidence limits are model-based limits from irr, not tests of corrected-minus-original changes; ordinal, sparse data warrant caution.
Negative ICC estimates are retained. Constant identical ratings can give 100% exact agreement while both correlations are undefined; undefined coefficients are not set to 1.
Exact agreement = diagonal count / number of complete pairs. Polarity agreement compares no (0/1) versus yes (3/4).

INTERPRETATION
These analyses quantify agreement between repeated responses, not accuracy against an external truth criterion.
Higher polychoric correlation does not necessarily imply greater absolute or exact agreement.
If a correction procedure used the duplicate responses themselves, its improvement is not independent validation.
The three main-text examples are deliberately illustrative. Use the all-pair tables/Supplement to describe the full distribution of changes.
Do not treat the three pairings from a triplicated question as independent replications.

REFERENCES / SOFTWARE DOCUMENTATION
https://cran.r-project.org/web/packages/polycor/refman/polycor.html
https://search.r-project.org/CRAN/refmans/irr/html/icc.html
Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.

MAIN-TEXT FIGURE CAPTION
Agreement between repeated ratings of three illustrative fragrance items, before (left) and after (right) correction.
Each matrix uses all complete pairs in its dataset; sample sizes may differ between datasets.
Rows and columns represent the earlier and later occurrence, respectively. Cells report counts and percentages of the matrix's complete pairs, not row percentages.
The four categories are 0 (definitely no), 1 (probably no), 3 (probably yes), and 4 (definitely yes). Outlined diagonal cells indicate exact agreement.
Polychoric correlations use the two_step estimator with no continuity correction. ICC(A,1) is the two-way, absolute-agreement, single-measurement intraclass correlation on the original 0,1,3,4 scores.
Items were selected after examining the data to illustrate different patterns, not as a random or unbiased sample of the repeated items.
A: Does.this.fragrance.have.vetiver.notes. [P14]. User-selected example 1.
B: Is.musk.a.leading.note.in.this.fragrance. [P11]. User-selected example 2.
C: Does.this.fragrance.cost.more.than..100.per.ounce. [P20]. User-selected example 3.
If present, NA indicates that a statistic is not estimable, and * flags a polychoric estimate near its numerical boundary.

SUPPLEMENT FIGURE CAPTION
Confusion matrices for all 31 item pairs (29 distinct repeated questions).
Original and corrected matrices are adjacent. In Supplement_all_available, each matrix uses every complete pair available in that file.
Supplement_matched_complete instead holds the fragrance sample fixed within each original/corrected comparison, matching the default main-text analysis.
For questions administered three times, all three pairwise comparisons are shown under the default all_pairs setting. These comparisons are not statistically independent.
Cells contain counts; shading is based on the fraction of all complete pairs and uses one fixed linear scale across every matrix.
The occurrence numbers above each pair identify rows versus columns. Category order is 0,1,3,4, left to right and top to bottom.
r = polychoric correlation; ICC = ICC(A,1), two-way absolute agreement for a single measurement. Exact = exact agreement. NA = not estimable; * = near-boundary polychoric estimate.
