Salesforce.com: Large Data volumes
In LDV cases, first and foremost you should consider options for Data
Reduction such as archiving, UI mashups etc.. Where LDV is unavoidable
the concept of Selectivity should be applied to the functional areas
impacted most.
Data Reduction Considerations:
Archiving :
Consider off-platform archiving solutions
Data Warehouse:
Consider a data warehouse for analytics
Mashups:
Leave data in-situ in external systems and integrate at the UI level
Selectivity Considerations:
Selective
Reduce the number of objects and fields used in a query.
Narrow Range Reporting
Apply selective report filtering.
Filtering:
Apply restrictive filtering to Listviews and SOQL where clauses.
Custom Indexes:
Can be effective where query values in the indexed field represent less than 10% (or <300K) of the overall record count.
Skinny Tables:
Can be effective as a report accelerator at 5M records plus.
SOQL – avoid NULL values (can’t be indexed)
Horizontal partitioning of data :
Split objects by geography for example.
Denormalisation:
To remove expensive joins use Apex Triggers to resolve FK
Divisions:
Acts like a DB partition (by geography, ownership etc.)
Avoid over parenting:
10K limit for child records, per parent record. For example avoid one parent account having more than 10,000 contacts etc.
Data Skew :
look for even distribution.
Loading Data :
Use the Bulk API to load large data volumes, via the Apex Data Loader perhaps (requires setting value to be explicitly set). The Bulk API, in simple terms, uploads the data into temporary tables then executes processing of the data (actual load into target objects) using parallel asynchronous processes. This offers potential performance improvements over the serial and synchronous combined upload+process model employed by all other loading techniques.
Data Loading Considerations;
Defer complex sharing rules
Disable Triggers and Workflow (post-process via Batch Apex)
Speed of operation; Insert then Update then Upsert (involves implicit query)
Group and sequence data to avoid parent record locking
Remember database statistics calculate overnight, wait to do performance testing
Tune the batch size (HTTP keepalives, GZIP compression)
Salesforce Large Data volume Best Practices
Source : http://www.saasanalogy.com/salesforce-com-large-data-volumes/
Data Reduction Considerations:
Archiving :
Consider off-platform archiving solutions
Data Warehouse:
Consider a data warehouse for analytics
Mashups:
Leave data in-situ in external systems and integrate at the UI level
Selectivity Considerations:
Selective
Reduce the number of objects and fields used in a query.
Narrow Range Reporting
Apply selective report filtering.
Filtering:
Apply restrictive filtering to Listviews and SOQL where clauses.
Custom Indexes:
Can be effective where query values in the indexed field represent less than 10% (or <300K) of the overall record count.
Skinny Tables:
Can be effective as a report accelerator at 5M records plus.
SOQL – avoid NULL values (can’t be indexed)
Horizontal partitioning of data :
Split objects by geography for example.
Denormalisation:
To remove expensive joins use Apex Triggers to resolve FK
Divisions:
Acts like a DB partition (by geography, ownership etc.)
Avoid over parenting:
Data Skew :
look for even distribution.
Loading Data :
Use the Bulk API to load large data volumes, via the Apex Data Loader perhaps (requires setting value to be explicitly set). The Bulk API, in simple terms, uploads the data into temporary tables then executes processing of the data (actual load into target objects) using parallel asynchronous processes. This offers potential performance improvements over the serial and synchronous combined upload+process model employed by all other loading techniques.
Data Loading Considerations;
Defer complex sharing rules
Disable Triggers and Workflow (post-process via Batch Apex)
Speed of operation; Insert then Update then Upsert (involves implicit query)
Group and sequence data to avoid parent record locking
Remember database statistics calculate overnight, wait to do performance testing
Tune the batch size (HTTP keepalives, GZIP compression)
Salesforce Large Data volume Best Practices
Source : http://www.saasanalogy.com/salesforce-com-large-data-volumes/
Comments
Post a Comment