实际情况是这样,有一个历史遗留系统,采用的数据库是oracle,我做的主要事情是将原来的数据适配到新的系统中来,为此我做了一个数据同步程序,读取oracle的数据,同步到新的msssql数据库中。
新的系统采用Entity Framework框架,这个时候问题出现了,原来oracle中有几张特别大的表,数据量大概在50w行以上,这样在我新系统中同步数据的时候,开始我使用的是develop webdev,出现out of memory,然后我又用本地iis调试,并且将应用程序池的回收机制改了,但是同步依然不断出现线程退出,具体原因未知。现在附上部分源代码,请各位帮忙分析分析ef如何存取海量数据。
- C# code
private void SynchronizeDailyMaintenanceModel() { this.EquipItemTable = this.EquipItemAdapter.GetData(); this.EquipItemVlTable = this.EquipItemVlAdapter.GetData(); this.InspectTable = this.InspectAdapter.GetData(); this.InspectItemVlTable = this.InspectItemVlAdapter.GetData(); DateTime maxLogTime; if (this.MmsEntityModel.DataSynchronizationLogEntities.Count() > 0) { maxLogTime = this.MmsEntityModel.DataSynchronizationLogEntities.Select(p => p.Time).Max(); } else { maxLogTime = Convert.ToDateTime("01/01/0001"); } var inspectQuery = from inspect in this.InspectTable where inspect.STANDARD_TIME > maxLogTime select inspect; foreach (MmsDataBase.NMS_INSPECTRow inspectRow in inspectQuery) { var inspectItemVlQuery = from inspectItemVl in this.InspectItemVlTable where inspectItemVl.INSPECT_ID == inspectRow.ID select inspectItemVl; foreach (MmsDataBase.NMS_INSPECT_ITEM_VLRow inspectItemVlRow in inspectItemVlQuery) { var equipItemVlQuery = from equipItemVl in this.EquipItemVlTable where equipItemVl.ID == inspectItemVlRow.ITEM_VL_ID select equipItemVl; foreach (MmsDataBase.NMS_EQUIP_ITEM_VLRow equipItemVlRow in equipItemVlQuery) { var equipItemQuery = from equipItem in this.EquipItemTable where equipItem.ID == equipItemVlRow.ITEM_ID select equipItem; foreach (MmsDataBase.NMS_EQUIP_ITEMRow equipItemRow in equipItemQuery) { var equipQuery = from equip in this.MmsEntityModel.EquipmentEntities where equip.ID == equipItemRow.EQUIP_ID select equip; if (equipQuery.Count() > 0) { DailyMaintenance maintenance = new DailyMaintenance(); maintenance.ID = inspectRow.ID; maintenance.Description = equipItemRow.NAME; maintenance.Time = inspectRow.STANDARD_TIME; maintenance.Cost = 0; Equipment parentEquip = equipQuery.First(); parentEquip.Maintenance.Add(maintenance); this.MmsEntityModel.MaintenanceEntities.AddObject(maintenance); } } } } } }
上面几个Table都是使用数据集的Table,为了将数据独到内存中来查询速度快。
在整个函数运行完了之后,再运行一下函数:
- C# code
private void SynchronizeMaintenanceModel() { this.SynchronizeDailyMaintenanceModel(); this.MmsEntityModel.SaveChanges(); }
------解决方案--------------------------------------------------------