Mongodb互联网金融的应用

1、Mongodb 在互联网金融的应用AgendaAbout KoalaWhy ? RequirementsHow ? ArchitectureProblemsSolutionsFutureWhat is P2PWhat is a P2P?P2P PlatformBorrowerLenderLenderLenderSingle LoanNoteNoteNoteLender :有钱人/投资人, Borrower:缺钱人/借款人 Note: 投资(凭证) Loan:借款标的(需求)Investing in multiple P2P?P2P PlatformBorrowerLenderLenderLen

2、derP2P PlatformBorrowerP2P PlatformBorrowerMultiple Loans Multiple Loans Multiple Loans Lender :有钱人/投资人, Borrower:缺钱人/借款人 Note: 投资(凭证) Loan:借款标的(需求)What is a P2P fund?P2P PlatformBorrowerLenderLenderLenderP2P PlatformBorrowerP2P PlatformBorrowerP2PFundMultiple LoansMultiple LoansMultiple LoansLender

3、 :有钱人/投资人, Borrower:缺钱人/借款人 Note: 投资(凭证) Loan:借款标的(需求) Fund:基金What is a P2P fund?P2P PlatformBorrowerLenderLenderLenderP2P PlatformBorrowerP2P PlatformBorrower考拉 理财Multiple LoansMultiple LoansMultiple LoansLender :有钱人/投资人, Borrower:缺钱人/借款人 Note: 投资(凭证) Loan:借款标的(需求) Fund:基金About KoalaTags: Finance ,

4、 P2P mashup，懒人 -通过极致的分散理财，达到风险&收益的平衡 正式上线：2015/5交易额：51+亿￥AUM：8+亿￥documents count: 千万 亿用户量：75万，同时在线: 1002,000About Koala投资策略 和管理系统风险监控模型和管理系统资产配置模型（源于高盛）基金财务系统用户管理系统System architectureTech StackMongodb(core) + Hadoop MySQLNodejs(Typescript), Python, R, Sql, Java(non core)Mongodb supports most of our

5、core systemsWhy? - Requirements for the marketLoans 稀疏多变Borrower 征信信息P2P platforms 稀疏多变市场需求（哪有不多变的需求）数据量多、N多维度统计要响应快Why? - Requirements for finance修改历史需要被保留 (不允许删除操作)操作要提供反操作，而不仅仅是简单的update备份，备份，备份安全演练Mongodb deploy architectureV1Mongodb deploy architectureV2Mongodb deploy architectureV3Mongodb depl

6、oy architectureV4Mongodb deploy architectureV5坑s:Ops & DevIssues ever met in opsIndex 不中，扫描过多比较耗内存Dev误操作程序连错db篡改数据误操作误删数据Issues ever solvedIndex 不中，扫描过多 - explain it + 慢查询分析: ensureindex比较耗内存 - 加内存吧，便宜的解决方案Dev误操作程序连错db篡改数据 - VPC & 区别开环境 & 做好权限控制误操作误删数据 - 延时节点！Oplog 恢复Security Summary角色限制: Administra

7、tor/Monitor/Read Write/Read Only不要用默认端口设置password给不同的password 区分开prod & dev env开发环境不允许连prod envBackup SummaryOplog 演练恢复到指定时间点Full backup Every 6 hoursDaily Backup within Recently 30 DaysDelay instance (1 hour) - 防止 replica set 实时污染第二个 DataCenter instanceIssues ever met in dev读写没分离压力 primary过大离线计算ETL

8、导致压力过大嵌套文档的数组长度过大“Schemaless”Issue ever solved in dev读写没分离压力 primary过大 - 读写分离离线计算ETL导致压力过大 - 读写分离 & 同步多一个DB用于计算嵌套文档的数组长度过大 - 设计注意考虑是否是不断增长、变化应用层or DB层限制Schema坑 s:part 2 Problems about TradingCrash while processingTransfer exampleA account 1000transfer 500￥B account 100Transfer example简单场景: account &

9、 trade are in the same DB system复杂场景: account & trade are in different service & DB System实际场景: Different Banks/SystemNeed Transaction?强一致性弱一致性Mongodb SolutionsEmbed documents优: Sometimes it works & make query faster缺: Sometimes it just doesnt workTwo phase commitMake code looks complicatedOther Sol

10、utionsMySQLMySQL (trade + account service) + Mongodb (other)PostgreSQLCassandraSequoiaDBHybrid alreadyRedis + MongodbRedis + MySQLHybrid Solution for Mysql & MongodbHybrid Solution ProblemsMaintainability: More ComplexForeign KeyJoin TablesSyncOthers Mongodb Only Solution: Events SourcingEvents Sour

11、cing : record every events and replay them after crashTrade offOur Solutions基于Embed docs 设计应用层(ORM) level:dev造了个轮子：mongoose-transact/mongoose-transtion-plugin/waterline-transactHow Transaction Is ImplementHow Transaction Is Implement in CodeUsage Sample CodeBest Practise for Account Trading(Common)可重入函数掉单检测:定时检查&重试单元测试注：支付异步回调会发生多次直到处理成功(限