Corrupt input files in literature project

A number of the files generated by the incremental literature pipeline include corrupt records.

The literature pipeline generates two sets of outputs, Abstracts and Full-text: both exhibit the same problem.

The following files are corrupt:

Full-text:

/Full-text/29_06_2022/NMP_patch-28-06-2022-8.jsonl /Full-text/14_06_2022/NMP_patch-13-06-2022-8.jsonl /Full-text/30_03_2022/NMP_patch-29-03-2022-2.jsonl /Full-text/18_06_2022/NMP_patch-17-06-2022-2.jsonl /Full-text/09_06_2022/NMP_patch-08-06-2022-5.jsonl /Full-text/28_06_2022/NMP_patch-27-06-2022-3.jsonl /Full-text/21_06_2022/NMP_patch-20-06-2022-6.jsonl /Full-text/27_06_2022/NMP_patch-26-06-2022-3.jsonl /Full-text/30_04_2022/NMP_patch-29-04-2022-8.jsonl /Full-text/13_04_2022/NMP_patch-12-04-2022-1.jsonl /Full-text/15_06_2022/NMP_patch-14-06-2022-12.jsonl /Full-text/26_05_2022/NMP_patch-25-05-2022-7.jsonl /Full-text/26_03_2022/NMP_patch-25-03-2022-3.jsonl /Full-text/29_04_2022/NMP_patch-28-04-2022-8.jsonl /Full-text/12_04_2022/NMP_patch-11-04-2022-1.jsonl /Full-text/16_06_2022/NMP_patch-15-06-2022-1.jsonl /Full-text/18_06_2022/NMP_patch-17-06-2022-0.jsonl /Full-text/18_04_2022/NMP_patch-17-04-2022-0.jsonl /Full-text/16_04_2022/NMP_patch-15-04-2022-9.jsonl /Full-text/14_06_2022/NMP_patch-13-06-2022-10.jsonl /Full-text/13_06_2022/NMP_patch-12-06-2022-1.jsonl /Full-text/30_04_2022/NMP_patch-29-04-2022-3.jsonl /Full-text/14_04_2022/NMP_patch-13-04-2022-1.jsonl /Full-text/16_04_2022/NMP_patch-15-04-2022-11.jsonl /Full-text/30_04_2022/NMP_patch-29-04-2022-1.jsonl /Full-text/17_05_2022/NMP_patch-16-05-2022-0.jsonl /Full-text/29_04_2022/NMP_patch-28-04-2022-3.jsonl /Full-text/13_05_2022/NMP_patch-12-05-2022-1.jsonl /Full-text/29_06_2022/NMP_patch-28-06-2022-7.jsonl /Full-text/15_04_2022/NMP_patch-14-04-2022-2.jsonl /Full-text/13_06_2022/NMP_patch-12-06-2022-9.jsonl /Full-text/27_03_2022/NMP_patch-26-03-2022-1.jsonl /Full-text/29_04_2022/NMP_patch-28-04-2022-7.jsonl /Full-text/12_05_2022/NMP_patch-11-05-2022-10.jsonl /Full-text/13_05_2022/NMP_patch-12-05-2022-8.jsonl /Full-text/17_05_2022/NMP_patch-16-05-2022-2.jsonl /Full-text/29_03_2022/NMP_patch-28-03-2022-3.jsonl /Full-text/19_04_2022/NMP_patch-18-04-2022-0.jsonl /Full-text/15_06_2022/NMP_patch-14-06-2022-9.jsonl /Full-text/12_04_2022/NMP_patch-11-04-2022-0.jsonl /Full-text/14_05_2022/NMP_patch-13-05-2022-1.jsonl /Full-text/10_06_2022/NMP_patch-09-06-2022-4.jsonl /Full-text/16_05_2022/NMP_patch-15-05-2022-1.jsonl /Full-text/12_07_2022/NMP_patch-11-07-2022-1.jsonl /Full-text/15_06_2022/NMP_patch-14-06-2022-11.jsonl /Full-text/27_03_2022/NMP_patch-26-03-2022-2.jsonl /Full-text/10_06_2022/NMP_patch-09-06-2022-2.jsonl /Full-text/15_04_2022/NMP_patch-14-04-2022-9.jsonl /Full-text/30_03_2022/NMP_patch-29-03-2022-1.jsonl /Full-text/17_04_2022/NMP_patch-16-04-2022-3.jsonl /Full-text/12_04_2022/NMP_patch-11-04-2022-8.jsonl /Full-text/20_04_2022/NMP_patch-19-04-2022-1.jsonl /Full-text/15_08_2022/NMP_patch-14-08-2022-23.jsonl /Full-text/17_06_2022/NMP_patch-16-06-2022-7.jsonl /Full-text/13_09_2022/NMP_patch-12-09-2022-20.jsonl /Full-text/28_07_2022/NMP_patch-27-07-2022-8.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-0.jsonl /Full-text/30_07_2022/NMP_patch-29-07-2022-17.jsonl /Full-text/25_07_2022/NMP_patch-24-07-2022-6.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-3.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-9.jsonl /Full-text/22_06_2022/NMP_patch-21-06-2022-7.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-6.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-7.jsonl /Full-text/20_07_2022/NMP_patch-19-07-2022-9.jsonl /Full-text/14_08_2022/NMP_patch-13-08-2022-5.jsonl /Full-text/12_09_2022/NMP_patch-11-09-2022-21.jsonl /Full-text/30_03_2022/NMP_patch-29-03-2022-8.jsonl /Full-text/07_09_2022/NMP_patch-06-09-2022-12.jsonl /Full-text/06_09_2022/NMP_patch-05-09-2022-10.jsonl /Full-text/26_05_2022/NMP_patch-25-05-2022-9.jsonl /Full-text/30_04_2022/NMP_patch-29-04-2022-9.jsonl /Full-text/29_04_2022/NMP_patch-28-04-2022-9.jsonl /Full-text/29_06_2022/NMP_patch-28-06-2022-9.jsonl /Full-text/10_06_2022/NMP_patch-09-06-2022-5.jsonl /Full-text/18_09_2022/NMP_patch-17-09-2022-1.jsonl /Full-text/14_09_2022/NMP_patch-13-09-2022-2.jsonl /Full-text/31_03_2022/NMP_patch-30-03-2022-0.jsonl /Full-text/23_03_2022/NMP_patch-total-300.jsonl /Full-text/23_03_2022/NMP_patch-total-63.jsonl /Full-text/23_03_2022/NMP_patch-total-2375.jsonl /Full-text/23_03_2022/NMP_patch-total-3545.jsonl /Full-text/23_03_2022/NMP_patch-total-750.jsonl /Full-text/23_03_2022/NMP_patch-total-2470.jsonl /Full-text/23_03_2022/NMP_patch-total-2599.jsonl /Full-text/23_03_2022/NMP_patch-total-2940.jsonl /Full-text/23_03_2022/NMP_patch-total-1882.jsonl /Full-text/23_03_2022/NMP_patch-total-264.jsonl /Full-text/23_03_2022/NMP_patch-total-2788.jsonl /Full-text/23_03_2022/NMP_patch-total-937.jsonl /Full-text/23_03_2022/NMP_patch-total-3059.jsonl /Full-text/23_03_2022/NMP_patch-total-3261.jsonl /Full-text/23_03_2022/NMP_patch-total-3378.jsonl /Full-text/23_03_2022/NMP_patch-total-3256.jsonl /Full-text/23_03_2022/NMP_patch-total-1740.jsonl /Full-text/23_03_2022/NMP_patch-total-3479.jsonl /Full-text/23_03_2022/NMP_patch-total-374.jsonl /Full-text/23_03_2022/NMP_patch-total-3128.jsonl /Full-text/23_03_2022/NMP_patch-total-444.jsonl /Full-text/23_03_2022/NMP_patch-total-3391.jsonl /Full-text/23_03_2022/NMP_patch-total-334.jsonl /Full-text/23_03_2022/NMP_patch-total-964.jsonl /Full-text/23_03_2022/NMP_patch-total-3248.jsonl /Full-text/23_03_2022/NMP_patch-total-480.jsonl /Full-text/23_03_2022/NMP_patch-total-608.jsonl /Full-text/23_03_2022/NMP_patch-total-2890.jsonl /Full-text/23_03_2022/NMP_patch-total-2659.jsonl /Full-text/23_03_2022/NMP_patch-total-850.jsonl /Full-text/23_03_2022/NMP_patch-total-399.jsonl /Full-text/23_03_2022/NMP_patch-total-100.jsonl /Full-text/23_03_2022/NMP_patch-total-2910.jsonl /Full-text/23_03_2022/NMP_patch-total-335.jsonl /Full-text/23_03_2022/NMP_patch-total-208.jsonl /Full-text/23_03_2022/NMP_patch-total-2900.jsonl /Full-text/23_03_2022/NMP_patch-total-2939.jsonl /Full-text/23_03_2022/NMP_patch-total-3082.jsonl /Full-text/23_03_2022/NMP_patch-total-545.jsonl /Full-text/23_03_2022/NMP_patch-total-3121.jsonl /Full-text/23_03_2022/NMP_patch-total-740.jsonl /Full-text/23_03_2022/NMP_patch-total-2385.jsonl /Full-text/23_03_2022/NMP_patch-total-599.jsonl /Full-text/23_03_2022/NMP_patch-total-3114.jsonl /Full-text/23_03_2022/NMP_patch-total-1752.jsonl /Full-text/23_03_2022/NMP_patch-total-255.jsonl /Full-text/23_03_2022/NMP_patch-total-2310.jsonl /Full-text/23_03_2022/NMP_patch-total-760.jsonl /Full-text/23_03_2022/NMP_patch-total-3527.jsonl /Full-text/23_03_2022/NMP_patch-total-3046.jsonl /Full-text/23_03_2022/NMP_patch-total-3075.jsonl /Full-text/23_03_2022/NMP_patch-total-453.jsonl /Full-text/23_03_2022/NMP_patch-total-2703.jsonl /Full-text/23_03_2022/NMP_patch-total-3482.jsonl /Full-text/23_03_2022/NMP_patch-total-3251.jsonl /Full-text/23_03_2022/NMP_patch-total-3505.jsonl /Full-text/23_03_2022/NMP_patch-total-965.jsonl /Full-text/23_03_2022/NMP_patch-total-3240.jsonl /Full-text/23_03_2022/NMP_patch-total-58.jsonl /Full-text/23_03_2022/NMP_patch-total-2228.jsonl /Full-text/23_03_2022/NMP_patch-total-474.jsonl /Full-text/23_03_2022/NMP_patch-total-69.jsonl /Full-text/23_03_2022/NMP_patch-total-898.jsonl /Full-text/23_03_2022/NMP_patch-total-2424.jsonl /Full-text/23_03_2022/NMP_patch-total-3139.jsonl /Full-text/23_03_2022/NMP_patch-total-2400.jsonl /Full-text/23_03_2022/NMP_patch-total-734.jsonl /Full-text/23_03_2022/NMP_patch-total-835.jsonl /Full-text/23_03_2022/NMP_patch-total-2795.jsonl /Full-text/23_03_2022/NMP_patch-total-3118.jsonl /Full-text/23_03_2022/NMP_patch-total-3117.jsonl /Full-text/23_03_2022/NMP_patch-total-2320.jsonl /Full-text/23_03_2022/NMP_patch-total-1717.jsonl /Full-text/23_03_2022/NMP_patch-total-773.jsonl /Full-text/23_03_2022/NMP_patch-total-3375.jsonl /Full-text/23_03_2022/NMP_patch-total-279.jsonl /Full-text/23_03_2022/NMP_patch-total-2907.jsonl /Full-text/23_03_2022/NMP_patch-total-3503.jsonl /Full-text/23_03_2022/NMP_patch-total-273.jsonl /Full-text/23_03_2022/NMP_patch-total-2997.jsonl /Full-text/23_03_2022/NMP_patch-total-549.jsonl /Full-text/23_03_2022/NMP_patch-total-2748.jsonl /Full-text/23_03_2022/NMP_patch-total-3064.jsonl /Full-text/23_03_2022/NMP_patch-total-2208.jsonl /Full-text/23_03_2022/NMP_patch-total-785.jsonl /Full-text/23_03_2022/NMP_patch-total-2893.jsonl /Full-text/23_03_2022/NMP_patch-total-263.jsonl /Full-text/23_03_2022/NMP_patch-total-3249.jsonl /Full-text/23_03_2022/NMP_patch-total-2888.jsonl /Full-text/23_03_2022/NMP_patch-total-2490.jsonl /Full-text/23_03_2022/NMP_patch-total-3141.jsonl /Full-text/23_03_2022/NMP_patch-total-625.jsonl /Full-text/23_03_2022/NMP_patch-total-3502.jsonl /Full-text/23_03_2022/NMP_patch-total-3522.jsonl /Full-text/23_03_2022/NMP_patch-total-179.jsonl /Full-text/23_03_2022/NMP_patch-total-500.jsonl /Full-text/23_03_2022/NMP_patch-total-3309.jsonl /Full-text/23_03_2022/NMP_patch-total-3057.jsonl /Full-text/23_03_2022/NMP_patch-total-2839.jsonl /Full-text/23_03_2022/NMP_patch-total-3254.jsonl /Full-text/23_03_2022/NMP_patch-total-3243.jsonl /Full-text/23_03_2022/NMP_patch-total-3523.jsonl /Full-text/23_03_2022/NMP_patch-total-834.jsonl /Full-text/23_03_2022/NMP_patch-total-2339.jsonl /Full-text/23_03_2022/NMP_patch-total-2829.jsonl /Full-text/23_03_2022/NMP_patch-total-190.jsonl /Full-text/23_03_2022/NMP_patch-total-445.jsonl /Full-text/23_03_2022/NMP_patch-total-1745.jsonl /Full-text/23_03_2022/NMP_patch-total-330.jsonl /Full-text/23_03_2022/NMP_patch-total-875.jsonl /Full-text/23_03_2022/NMP_patch-total-710.jsonl /Full-text/23_03_2022/NMP_patch-total-2359.jsonl /Full-text/23_03_2022/NMP_patch-total-2384.jsonl /Full-text/23_03_2022/NMP_patch-total-2554.jsonl /Full-text/23_03_2022/NMP_patch-total-2830.jsonl /Full-text/23_03_2022/NMP_patch-total-2488.jsonl /Full-text/23_03_2022/NMP_patch-total-870.jsonl /Full-text/23_03_2022/NMP_patch-total-2823.jsonl /Full-text/23_03_2022/NMP_patch-total-775.jsonl /Full-text/23_03_2022/NMP_patch-total-2535.jsonl /Full-text/23_03_2022/NMP_patch-total-295.jsonl /Full-text/23_03_2022/NMP_patch-total-3373.jsonl /Full-text/23_03_2022/NMP_patch-total-3123.jsonl /Full-text/23_03_2022/NMP_patch-total-2922.jsonl /Full-text/23_03_2022/NMP_patch-total-2449.jsonl /Full-text/23_03_2022/NMP_patch-total-3120.jsonl /Full-text/23_03_2022/NMP_patch-total-830.jsonl /Full-text/23_03_2022/NMP_patch-total-3221.jsonl /Full-text/23_03_2022/NMP_patch-total-3132.jsonl /Full-text/23_03_2022/NMP_patch-total-3371.jsonl /Full-text/23_03_2022/NMP_patch-total-493.jsonl /Full-text/23_03_2022/NMP_patch-total-2464.jsonl /Full-text/23_03_2022/NMP_patch-total-2589.jsonl /Full-text/23_03_2022/NMP_patch-total-3124.jsonl /Full-text/23_03_2022/NMP_patch-total-3478.jsonl /Full-text/23_03_2022/NMP_patch-total-1770.jsonl /Full-text/23_03_2022/NMP_patch-total-909.jsonl /Full-text/23_03_2022/NMP_patch-total-3501.jsonl /Full-text/23_03_2022/NMP_patch-total-3526.jsonl /Full-text/23_03_2022/NMP_patch-total-2332.jsonl /Full-text/23_03_2022/NMP_patch-total-3259.jsonl /Full-text/23_03_2022/NMP_patch-total-3165.jsonl /Full-text/23_03_2022/NMP_patch-total-3247.jsonl /Full-text/23_03_2022/NMP_patch-total-2909.jsonl /Full-text/23_03_2022/NMP_patch-total-2564.jsonl /Full-text/23_03_2022/NMP_patch-total-3386.jsonl /Full-text/23_03_2022/NMP_patch-total-3252.jsonl /Full-text/23_03_2022/NMP_patch-total-3381.jsonl /Full-text/23_03_2022/NMP_patch-total-2802.jsonl /Full-text/23_03_2022/NMP_patch-total-590.jsonl /Full-text/23_03_2022/NMP_patch-total-3145.jsonl /Full-text/23_03_2022/NMP_patch-total-375.jsonl /Full-text/23_03_2022/NMP_patch-total-479.jsonl /Full-text/23_03_2022/NMP_patch-total-498.jsonl /Full-text/23_03_2022/NMP_patch-total-2944.jsonl /Full-text/23_03_2022/NMP_patch-total-475.jsonl /Full-text/23_03_2022/NMP_patch-total-3230.jsonl /Full-text/23_03_2022/NMP_patch-total-2899.jsonl /Full-text/23_03_2022/NMP_patch-total-2908.jsonl /Full-text/23_03_2022/NMP_patch-total-454.jsonl /Full-text/23_03_2022/NMP_patch-total-3389.jsonl /Full-text/23_03_2022/NMP_patch-total-3231.jsonl /Full-text/23_03_2022/NMP_patch-total-70.jsonl /Full-text/23_03_2022/NMP_patch-total-189.jsonl /Full-text/23_03_2022/NMP_patch-total-3122.jsonl /Full-text/23_03_2022/NMP_patch-total-2603.jsonl /Full-text/23_03_2022/NMP_patch-total-3524.jsonl /Full-text/23_03_2022/NMP_patch-total-2337.jsonl /Full-text/23_03_2022/NMP_patch-total-2075.jsonl /Full-text/23_03_2022/NMP_patch-total-613.jsonl /Full-text/23_03_2022/NMP_patch-total-3481.jsonl /Full-text/23_03_2022/NMP_patch-total-3250.jsonl /Full-text/23_03_2022/NMP_patch-total-3613.jsonl /Full-text/23_03_2022/NMP_patch-total-705.jsonl /Full-text/23_03_2022/NMP_patch-total-3480.jsonl /Full-text/23_03_2022/NMP_patch-total-3257.jsonl /Full-text/23_03_2022/NMP_patch-total-2338.jsonl /Full-text/23_03_2022/NMP_patch-total-149.jsonl /Full-text/23_03_2022/NMP_patch-total-325.jsonl /Full-text/23_03_2022/NMP_patch-total-3053.jsonl /Full-text/23_03_2022/NMP_patch-total-3516.jsonl /Full-text/23_03_2022/NMP_patch-total-380.jsonl /Full-text/23_03_2022/NMP_patch-total-3380.jsonl /Full-text/23_03_2022/NMP_patch-total-478.jsonl /Full-text/23_03_2022/NMP_patch-total-1009.jsonl /Full-text/23_03_2022/NMP_patch-total-2897.jsonl /Full-text/23_03_2022/NMP_patch-total-524.jsonl /Full-text/23_03_2022/NMP_patch-total-3372.jsonl /Full-text/23_03_2022/NMP_patch-total-194.jsonl /Full-text/23_03_2022/NMP_patch-total-764.jsonl /Full-text/23_03_2022/NMP_patch-total-3223.jsonl /Full-text/23_03_2022/NMP_patch-total-890.jsonl /Full-text/23_03_2022/NMP_patch-total-3133.jsonl /Full-text/23_03_2022/NMP_patch-total-869.jsonl /Full-text/23_03_2022/NMP_patch-total-935.jsonl /Full-text/23_03_2022/NMP_patch-total-3228.jsonl /Full-text/23_03_2022/NMP_patch-total-720.jsonl /Full-text/23_03_2022/NMP_patch-total-3142.jsonl /Full-text/23_03_2022/NMP_patch-total-3513.jsonl /Full-text/23_03_2022/NMP_patch-total-3055.jsonl /Full-text/23_03_2022/NMP_patch-total-845.jsonl /Full-text/23_03_2022/NMP_patch-total-848.jsonl /Full-text/23_03_2022/NMP_patch-total-3507.jsonl /Full-text/23_03_2022/NMP_patch-total-2825.jsonl /Full-text/23_03_2022/NMP_patch-total-3258.jsonl /Full-text/23_03_2022/NMP_patch-total-288.jsonl /Full-text/23_03_2022/NMP_patch-total-2550.jsonl /Full-text/23_03_2022/NMP_patch-total-3222.jsonl /Full-text/23_03_2022/NMP_patch-total-3236.jsonl /Full-text/23_03_2022/NMP_patch-total-3080.jsonl /Full-text/23_03_2022/NMP_patch-total-2340.jsonl /Full-text/23_03_2022/NMP_patch-total-970.jsonl /Full-text/23_03_2022/NMP_patch-total-65.jsonl /Full-text/23_03_2022/NMP_patch-total-2958.jsonl /Full-text/23_03_2022/NMP_patch-total-2873.jsonl /Full-text/23_03_2022/NMP_patch-total-370.jsonl /Full-text/23_03_2022/NMP_patch-total-3149.jsonl /Full-text/23_03_2022/NMP_patch-total-3037.jsonl /Full-text/23_03_2022/NMP_patch-total-860.jsonl /Full-text/23_03_2022/NMP_patch-total-3392.jsonl /Full-text/23_03_2022/NMP_patch-total-3148.jsonl /Full-text/23_03_2022/NMP_patch-total-3040.jsonl /Full-text/23_03_2022/NMP_patch-total-3152.jsonl /Full-text/23_03_2022/NMP_patch-total-2954.jsonl /Full-text/23_03_2022/NMP_patch-total-2850.jsonl /Full-text/23_03_2022/NMP_patch-total-3612.jsonl /Full-text/23_03_2022/NMP_patch-total-2710.jsonl /Full-text/23_03_2022/NMP_patch-total-355.jsonl /Full-text/23_03_2022/NMP_patch-total-3013.jsonl /Full-text/23_03_2022/NMP_patch-total-2858.jsonl /Full-text/23_03_2022/NMP_patch-total-3525.jsonl /Full-text/23_03_2022/NMP_patch-total-735.jsonl /Full-text/23_03_2022/NMP_patch-total-899.jsonl /Full-text/23_03_2022/NMP_patch-total-178.jsonl /Full-text/23_03_2022/NMP_patch-total-3356.jsonl /Full-text/23_03_2022/NMP_patch-total-2144.jsonl /Full-text/23_03_2022/NMP_patch-total-3239.jsonl /Full-text/23_03_2022/NMP_patch-total-3618.jsonl /Full-text/23_03_2022/NMP_patch-total-1020.jsonl /Full-text/23_03_2022/NMP_patch-total-3203.jsonl /Full-text/23_03_2022/NMP_patch-total-2824.jsonl /Full-text/23_03_2022/NMP_patch-total-3260.jsonl /Full-text/23_03_2022/NMP_patch-total-2270.jsonl /Full-text/23_03_2022/NMP_patch-total-3255.jsonl /Full-text/23_03_2022/NMP_patch-total-1829.jsonl /Full-text/23_03_2022/NMP_patch-total-3436.jsonl /Full-text/23_03_2022/NMP_patch-total-2805.jsonl /Full-text/23_03_2022/NMP_patch-total-784.jsonl /Full-text/23_03_2022/NMP_patch-total-175.jsonl /Full-text/23_03_2022/NMP_patch-total-310.jsonl /Full-text/23_03_2022/NMP_patch-total-650.jsonl /Full-text/23_03_2022/NMP_patch-total-649.jsonl /Full-text/23_03_2022/NMP_patch-total-2982.jsonl /Full-text/23_03_2022/NMP_patch-total-3528.jsonl /Full-text/23_03_2022/NMP_patch-total-3134.jsonl /Full-text/23_03_2022/NMP_patch-total-833.jsonl /Full-text/23_03_2022/NMP_patch-total-150.jsonl /Full-text/23_03_2022/NMP_patch-total-3151.jsonl /Full-text/23_03_2022/NMP_patch-total-3246.jsonl /Full-text/23_03_2022/NMP_patch-total-1890.jsonl /Full-text/23_03_2022/NMP_patch-total-844.jsonl /Full-text/23_03_2022/NMP_patch-total-645.jsonl /Full-text/23_03_2022/NMP_patch-total-1762.jsonl /Full-text/23_03_2022/NMP_patch-total-2450.jsonl /Full-text/23_03_2022/NMP_patch-total-840.jsonl /Full-text/23_03_2022/NMP_patch-total-3511.jsonl /Full-text/23_03_2022/NMP_patch-total-284.jsonl /Full-text/23_03_2022/NMP_patch-total-2877.jsonl /Full-text/23_03_2022/NMP_patch-total-639.jsonl /Full-text/23_03_2022/NMP_patch-total-3253.jsonl /Full-text/23_03_2022/NMP_patch-total-715.jsonl /Full-text/23_03_2022/NMP_patch-total-2905.jsonl /Full-text/23_03_2022/NMP_patch-total-3234.jsonl /Full-text/23_03_2022/NMP_patch-total-2892.jsonl /Full-text/23_03_2022/NMP_patch-total-3521.jsonl /Full-text/23_03_2022/NMP_patch-total-499.jsonl /Full-text/23_03_2022/NMP_patch-total-3224.jsonl /Full-text/23_03_2022/NMP_patch-total-398.jsonl /Full-text/23_03_2022/NMP_patch-total-3146.jsonl /Full-text/23_03_2022/NMP_patch-total-948.jsonl /Full-text/23_03_2022/NMP_patch-total-2800.jsonl /Full-text/23_03_2022/NMP_patch-total-3147.jsonl /Full-text/23_03_2022/NMP_patch-total-3136.jsonl /Full-text/23_03_2022/NMP_patch-total-2544.jsonl /Full-text/23_03_2022/NMP_patch-total-1910.jsonl /Full-text/23_03_2022/NMP_patch-total-2280.jsonl /Full-text/23_03_2022/NMP_patch-total-3153.jsonl /Full-text/23_03_2022/NMP_patch-total-2765.jsonl /Full-text/23_03_2022/NMP_patch-total-1893.jsonl /Full-text/23_03_2022/NMP_patch-total-3150.jsonl /Full-text/23_03_2022/NMP_patch-total-324.jsonl /Full-text/23_03_2022/NMP_patch-total-89.jsonl /Full-text/23_03_2022/NMP_patch-total-3232.jsonl /Full-text/23_03_2022/NMP_patch-total-533.jsonl /Full-text/23_03_2022/NMP_patch-total-3229.jsonl /Full-text/23_03_2022/NMP_patch-total-199.jsonl /Full-text/23_03_2022/NMP_patch-total-3135.jsonl /Full-text/23_03_2022/NMP_patch-total-315.jsonl /Full-text/23_03_2022/NMP_patch-total-3065.jsonl /Full-text/23_03_2022/NMP_patch-total-495.jsonl /Full-text/23_03_2022/NMP_patch-total-3054.jsonl /Full-text/23_03_2022/NMP_patch-total-2334.jsonl /Full-text/23_03_2022/NMP_patch-total-390.jsonl /Full-text/23_03_2022/NMP_patch-total-159.jsonl /Full-text/23_03_2022/NMP_patch-total-3050.jsonl /Full-text/23_03_2022/NMP_patch-total-3143.jsonl /Full-text/23_03_2022/NMP_patch-total-2425.jsonl /Full-text/23_03_2022/NMP_patch-total-979.jsonl /Full-text/23_03_2022/NMP_patch-total-3140.jsonl /Full-text/23_03_2022/NMP_patch-total-494.jsonl /Full-text/23_03_2022/NMP_patch-total-3508.jsonl /Full-text/23_03_2022/NMP_patch-total-2405.jsonl /Full-text/23_03_2022/NMP_patch-total-3144.jsonl /Full-text/23_03_2022/NMP_patch-total-455.jsonl /Full-text/23_03_2022/NMP_patch-total-2670.jsonl /Full-text/23_03_2022/NMP_patch-total-679.jsonl /Full-text/23_03_2022/NMP_patch-total-2717.jsonl /Full-text/23_03_2022/NMP_patch-total-2945.jsonl /Full-text/23_03_2022/NMP_patch-total-2864.jsonl /Full-text/23_03_2022/NMP_patch-total-2289.jsonl /Full-text/23_03_2022/NMP_patch-total-2784.jsonl /Full-text/23_03_2022/NMP_patch-total-3167.jsonl /Full-text/23_03_2022/NMP_patch-total-3512.jsonl /Full-text/23_03_2022/NMP_patch-total-2274.jsonl /Full-text/23_03_2022/NMP_patch-total-1037.jsonl /Full-text/23_03_2022/NMP_patch-total-2155.jsonl /Full-text/23_03_2022/NMP_patch-total-765.jsonl /Full-text/23_03_2022/NMP_patch-total-655.jsonl /Full-text/23_03_2022/NMP_patch-total-2963.jsonl /Full-text/23_03_2022/NMP_patch-total-674.jsonl /Full-text/23_03_2022/NMP_patch-total-2494.jsonl /Full-text/23_03_2022/NMP_patch-total-314.jsonl /Full-text/23_03_2022/NMP_patch-total-968.jsonl /Full-text/23_03_2022/NMP_patch-total-719.jsonl /Full-text/23_03_2022/NMP_patch-total-910.jsonl /Full-text/23_03_2022/NMP_patch-total-2474.jsonl /Full-text/23_03_2022/NMP_patch-total-3640.jsonl /Full-text/23_03_2022/NMP_patch-total-3008.jsonl /Full-text/23_03_2022/NMP_patch-total-2440.jsonl /Full-text/23_03_2022/NMP_patch-total-254.jsonl /Full-text/23_03_2022/NMP_patch-total-725.jsonl /Full-text/23_03_2022/NMP_patch-total-858.jsonl /Full-text/23_03_2022/NMP_patch-total-2833.jsonl /Full-text/23_03_2022/NMP_patch-total-595.jsonl /Full-text/23_03_2022/NMP_patch-total-2634.jsonl /Full-text/23_03_2022/NMP_patch-total-2178.jsonl /Full-text/23_03_2022/NMP_patch-total-3130.jsonl /Full-text/23_03_2022/NMP_patch-total-2665.jsonl /Full-text/23_03_2022/NMP_patch-total-1760.jsonl /Full-text/23_03_2022/NMP_patch-total-699.jsonl /Full-text/23_03_2022/NMP_patch-total-1695.jsonl /Full-text/23_03_2022/NMP_patch-total-2040.jsonl /Full-text/23_03_2022/NMP_patch-total-415.jsonl /Full-text/23_03_2022/NMP_patch-total-889.jsonl /Full-text/23_03_2022/NMP_patch-total-2914.jsonl /Full-text/23_03_2022/NMP_patch-total-2759.jsonl /Full-text/23_03_2022/NMP_patch-total-529.jsonl /Full-text/23_03_2022/NMP_patch-total-2510.jsonl /Full-text/23_03_2022/NMP_patch-total-3138.jsonl /Full-text/23_03_2022/NMP_patch-total-2904.jsonl /Full-text/23_03_2022/NMP_patch-total-85.jsonl /Full-text/23_03_2022/NMP_patch-total-960.jsonl /Full-text/23_03_2022/NMP_patch-total-943.jsonl /Full-text/23_03_2022/NMP_patch-total-504.jsonl /Full-text/23_03_2022/NMP_patch-total-993.jsonl /Full-text/23_03_2022/NMP_patch-total-1854.jsonl /Full-text/23_03_2022/NMP_patch-total-820.jsonl /Full-text/23_03_2022/NMP_patch-total-2848.jsonl /Full-text/23_03_2022/NMP_patch-total-3263.jsonl /Full-text/23_03_2022/NMP_patch-total-1825.jsonl /Full-text/23_03_2022/NMP_patch-total-585.jsonl /Full-text/23_03_2022/NMP_patch-total-2129.jsonl /Full-text/23_03_2022/NMP_patch-total-410.jsonl /Full-text/23_03_2022/NMP_patch-total-2333.jsonl /Full-text/23_03_2022/NMP_patch-total-379.jsonl /Full-text/23_03_2022/NMP_patch-total-2693.jsonl /Full-text/23_03_2022/NMP_patch-total-3023.jsonl /Full-text/23_03_2022/NMP_patch-total-3635.jsonl /Full-text/23_03_2022/NMP_patch-total-369.jsonl /Full-text/23_03_2022/NMP_patch-total-2595.jsonl /Full-text/23_03_2022/NMP_patch-total-2650.jsonl /Full-text/23_03_2022/NMP_patch-total-205.jsonl /Full-text/23_03_2022/NMP_patch-total-95.jsonl /Full-text/23_03_2022/NMP_patch-total-644.jsonl /Full-text/23_03_2022/NMP_patch-total-1859.jsonl /Full-text/23_03_2022/NMP_patch-total-2724.jsonl /Full-text/23_03_2022/NMP_patch-total-1064.jsonl /Full-text/23_03_2022/NMP_patch-total-2278.jsonl /Full-text/23_03_2022/NMP_patch-total-420.jsonl /Full-text/23_03_2022/NMP_patch-total-1002.jsonl /Full-text/23_03_2022/NMP_patch-total-3269.jsonl /Full-text/23_03_2022/NMP_patch-total-2745.jsonl /Full-text/23_03_2022/NMP_patch-total-863.jsonl /Full-text/23_03_2022/NMP_patch-total-2350.jsonl /Full-text/23_03_2022/NMP_patch-total-2390.jsonl /Full-text/23_03_2022/NMP_patch-total-2844.jsonl /Full-text/23_03_2022/NMP_patch-total-2448.jsonl /Full-text/23_03_2022/NMP_patch-total-2684.jsonl /Full-text/23_03_2022/NMP_patch-total-2064.jsonl /Full-text/23_03_2022/NMP_patch-total-2979.jsonl /Full-text/23_03_2022/NMP_patch-total-3045.jsonl /Full-text/23_03_2022/NMP_patch-total-570.jsonl /Full-text/23_03_2022/NMP_patch-total-2943.jsonl /Full-text/23_03_2022/NMP_patch-total-2515.jsonl /Full-text/23_03_2022/NMP_patch-total-414.jsonl /Full-text/23_03_2022/NMP_patch-total-2395.jsonl /Full-text/23_03_2022/NMP_patch-total-80.jsonl /Full-text/23_03_2022/NMP_patch-total-2689.jsonl /Full-text/23_03_2022/NMP_patch-total-759.jsonl /Full-text/23_03_2022/NMP_patch-total-730.jsonl /Full-text/23_03_2022/NMP_patch-total-389.jsonl /Full-text/23_03_2022/NMP_patch-total-739.jsonl /Full-text/23_03_2022/NMP_patch-total-2049.jsonl /Full-text/23_03_2022/NMP_patch-total-2568.jsonl /Full-text/23_03_2022/NMP_patch-total-304.jsonl /Full-text/23_03_2022/NMP_patch-total-3051.jsonl /Full-text/23_03_2022/NMP_patch-total-2640.jsonl /Full-text/23_03_2022/NMP_patch-total-2712.jsonl /Full-text/23_03_2022/NMP_patch-total-1965.jsonl /Full-text/23_03_2022/NMP_patch-total-435.jsonl /Full-text/23_03_2022/NMP_patch-total-2020.jsonl /Full-text/23_03_2022/NMP_patch-total-3005.jsonl /Full-text/23_03_2022/NMP_patch-total-2778.jsonl /Full-text/23_03_2022/NMP_patch-total-905.jsonl /Full-text/23_03_2022/NMP_patch-total-2629.jsonl /Full-text/23_03_2022/NMP_patch-total-793.jsonl /Full-text/23_03_2022/NMP_patch-total-3034.jsonl /Full-text/23_03_2022/NMP_patch-total-975.jsonl /Full-text/23_03_2022/NMP_patch-total-2233.jsonl /Full-text/23_03_2022/NMP_patch-total-470.jsonl /Full-text/23_03_2022/NMP_patch-total-240.jsonl /Full-text/23_03_2022/NMP_patch-total-154.jsonl /Full-text/23_03_2022/NMP_patch-total-2968.jsonl /Full-text/23_03_2022/NMP_patch-total-855.jsonl /Full-text/23_03_2022/NMP_patch-total-2810.jsonl /Full-text/23_03_2022/NMP_patch-total-2884.jsonl /Full-text/23_03_2022/NMP_patch-total-1820.jsonl /Full-text/23_03_2022/NMP_patch-total-2974.jsonl /Full-text/23_03_2022/NMP_patch-total-145.jsonl /Full-text/23_03_2022/NMP_patch-total-2175.jsonl /Full-text/23_03_2022/NMP_patch-total-2820.jsonl /Full-text/23_03_2022/NMP_patch-total-1000.jsonl /Full-text/23_03_2022/NMP_patch-total-2220.jsonl /Full-text/23_03_2022/NMP_patch-total-1804.jsonl /Full-text/23_03_2022/NMP_patch-total-225.jsonl /Full-text/23_03_2022/NMP_patch-total-744.jsonl /Full-text/23_03_2022/NMP_patch-total-2114.jsonl /Full-text/23_03_2022/NMP_patch-total-2058.jsonl /Full-text/23_03_2022/NMP_patch-total-2794.jsonl /Full-text/23_03_2022/NMP_patch-total-2574.jsonl /Full-text/23_03_2022/NMP_patch-total-1070.jsonl /Full-text/23_03_2022/NMP_patch-total-125.jsonl /Full-text/23_03_2022/NMP_patch-total-405.jsonl Abstracts:

Abstracts/13_08_2022/NMP_patch-12-08-2022-0.jsonl Abstracts/14_08_2022/NMP_patch-13-08-2022-0.jsonl Abstracts/15_08_2022/NMP_patch-14-08-2022-0.jsonl Abstracts/16_08_2022/NMP_patch-15-08-2022-0.jsonl Abstracts/17_08_2022/NMP_patch-16-08-2022-0.jsonl Abstracts/18_08_2022/NMP_patch-17-08-2022-0.jsonl Abstracts/19_08_2022/NMP_patch-18-08-2022-0.jsonl Abstracts/20_08_2022/NMP_patch-19-08-2022-0.jsonl Abstracts/22_08_2022/NMP_patch-21-08-2022-0.jsonl Abstracts/23_08_2022/NMP_patch-22-08-2022-0.jsonl Abstracts/24_08_2022/NMP_patch-23-08-2022-0.jsonl When these files are ingested into Spark we get a column called _corrupt_record.

Based on eye-ball investigation of the cause, it looks like the write process is interrupted mid-stream as the record abruptly terminates.

To Reproduce Steps to reproduce the behaviour:

Download one of the corrupt files Execute jq . Observer error messages