Answers for "import huge csv file and remove duplicates python"

Python

1

remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Posted by: Guest on June-13-2021

Source

Code answers related to "import huge csv file and remove duplicates python"

Code answers related to "Python"

Python Answers by Framework

Django
Flask

Browse Popular Code Answers by Language

Python

Javascript

Whatever

Shell/Bash

CSS

Html

PHP

SQL

Java

C#