Categories

See More
Popular Forum

MBA (4887) B.Tech (1769) Engineering (1486) Class 12 (1030) Study Abroad (1004) Computer Science and Engineering (988) Business Management Studies (865) BBA (846) Diploma (746) CAT (651) B.Com (648) B.Sc (643) JEE Mains (618) Mechanical Engineering (574) Exam (525) India (462) Career (452) All Time Q&A (439) Mass Communication (427) BCA (417) Science (384) Computers & IT (Non-Engg) (383) Medicine & Health Sciences (381) Hotel Management (373) Civil Engineering (353) MCA (349) Tuteehub Top Questions (348) Distance (340) Colleges in India (334)
See More

How to join two json files in python instead of nesting for loop

General Tech Bugs & Fixes

Max. 2000 characters
Replies

usr_profile.png

User

( 6 months ago )

 

Every time i will get 500 records form file1 to join file2 which contains more than 100000 records it costs two minutes !!

with open(file1,'r') as f1,open(file2,'r') as f2:
    a=json.load(f1)
    b=json.load(f2)
    list_a=[]
    for i in range(len(a)):
        for n in range(len(b)):
            if b[n]["id"]==a[i]["id"]:
                list_a.append(dict(b[n],**a[i]))
with open(result,'w') as f3:
    json.dump(list_a, f3,sort_keys=True, ensure_ascii=False)

File1:

[{ "id":"1", "name":"Tom" }, 
{ "id":"2", "name":"Jim" }, 
{ "id":"3", "name":"Bob" }, 
{ "id":"4", "name":"Jeny" },  
{ "id":"5", "name":"Lara" }, 
{ "id":"6", "name":"Lin" }, 
{ "id":"7", "name":"Kim" }, 
{ "id":"8", "name":"Jack" }, 
{ "id":"9", "name":"Tony" }]

File 2:

[ { "id":"1", "Details":[ { "label":"jcc", "hooby":"Swimming" }, { "label":"hkt", "hooby":"Basketball" }, ] }, 
{ "id":"2", "Details":[ { 
usr_profile.png

User

( 6 months ago )

 

I don't have the experience to know if this would speed it up. The solution below provided by Eugene Yarmash seems more reliable. I also don't have the big files to test speed, but you can try and see if using collections would speed up the iteration. I'd actually be curious myself if it would change anything:

File1 = [ { "id":"1", "name":"Tom" }, { "id":"2", "name":"Jim" }, { "id":"3", "name":"Bob" }, { "id":"4", "name":"Jeny" }, { "id":"5", "name":"Lara" }, { "id":"6", "name":"Lin" }, { "id":"7", "name":"Kim" }, { "id":"8", "name":"Jack" }, { "id":"9", "name":"Tony" } ]
File2 = [ { "id":"1", "Details":[ { "label":"jcc", "hooby":"Swimming" }, { "label":"hkt", "hooby":"Basketball" }, ] }, { "id":"2", "Details":[ { "label":"NTC", "hooby":"Games" } ] } ] 

from collections import defaultdict

d = defaultdict(dict)
for l in (File1, File2):
    for elem in l:
        d[elem['id']].update(elem)
Result = dict(d)

what's your interest