Big Objects allow for storage of massive amounts of data on the Salesforce platform. The biggest draw for Big Objects isthat they can provide consistent performance for objects with records upwards of a billion with a standard set of APIs within the org as well as an external system (REST, SOAP, Bulk etc).
There are two types of Big Objects:
Standard big objects - defined and included in standard salesforce products to support data requirements for certain features. Example of a standard big object is FieldHistoryArchive which salesforce uses to track upto a year's worth of Field History data as a part of the Field Audit Trailproduct. Useful for regulated industry data that is required to be archived.
Custom big objects - defined and deployed through the Metadata API. Metadata components required to define a custom object include:
An object file contains the object definition, fields and index. Similar to custom object files with minor differences specifically the index.
A permissionset (optional) to defined the object and field level permissions for the object. Existing permission sets can be used grant permissions (I didn't use the word access, because when I think access I think record level access and is not supported with Big Objects).
A package to deploy the metadata definitions and permission. Developers and people that have used the ant migration tool would be more than familiar with a package.
Considerations
Due to the size and scale of big objects, there are certain limitations that need to be considered when implementing them
Big objects support only object and field permissions. This is an important one if there is sharing restrictions based on roles around the object data.
You must use the Metadata API to define or add a field to a custom big object. You can’t do it through the UI. This one might not be as bad if admins aren't required to modify Big Object definitions.
SOQL relationship queries are based on a lookup field from a big object to a standard or custom object in the select field list (not in filters or subqueries).
Big objects support custom Salesforce Lightning and Visualforce components rather than standard UI elements (home pages, detail pages, list views, and so on). Another huge limitation if you want to expose Big Object data without customization.
You can create up to 100 big objects per org. The limits for big object fields are similar to the limits on custom objects, and depend on your org’s license type.
Big objects don’t support transactions that include big objects, standard objects, and custom objects.
To support the scale of data in a big object, you can’t use triggers, flows, processes, and the Salesforce app. This can be a huge limitation if the data in a big object is required to be manipulated due to business processes.
Defining a Big Object
Big Objects can only be defined through the Metadata API. The components required to define a big object as outlined above are an object file, permissionset (optional) file for object and field level security and a package file. See examples below:
Object File
Key differences from non big object files include
Lack of sharing tags. e.g. sharingmodel, externalsharingmodel, enablesharing etc.
Lack of standard visual elements metadata tags. e.g. listviews, sharinglayouts etc.
Simplified field definitions. e.g. Lack of field history tracking, no formula fields, no lookup fields or filters etc.
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<fields>
<fullName>Start_Location_Lat__c</fullName>
<label>Start Location Latitude</label>
<required>false</required>
<type>Number</type>
<unique>false</unique>
<scale>4</scale>
<precision>7</precision>
</fields>
<fields>
<fullName>Start_Location_Long__c</fullName>
<label>Start Location Longitude</label>
<required>false</required>
<type>Number</type>
<unique>false</unique>
<scale>4</scale>
<precision>7</precision>
</fields>
<fields>
<fullName>Start_Time__c</fullName>
<label>Start Time</label>
<required>True</required>
<type>DateTime</type>
<unique>false</unique>
</fields>
<fields>
<fullName>End_Time__c</fullName>
<label>End Time</label>
<required>false</required>
<type>DateTime</type>
<unique>false</unique>
</fields>
<fields>
<fullName>Service_Type__c</fullName>
<label>Service Type</label>
<length>16</length>
<required>false</required>
<type>Text</type>
<unique>false</unique>
</fields>
<fields>
<fullName>Rider_Account__c</fullName>
<label>Rider Account</label>
<length>16</length>
<required>true</required>
<type>Text</type>
<unique>false</unique>
</fields>
<fields>
<fullName>Rider_Rating__c</fullName>
<label>Rider Rating</label>
<required>false</required>
<type>Number</type>
<unique>false</unique>
<scale>1</scale>
<precision>2</precision>
</fields>
<indexes>
<fullName>Rider_History_Index</fullName>
<label>Rider History Index</label>
<fields>
<name>Rider_Account__c</name>
<sortDirection>DESC</sortDirection>
</fields>
<fields>
<name>Start_Time__c</name>
<sortDirection>ASC</sortDirection>
</fields>
</indexes>
<label>Rider History</label>
<pluralLabel>Rider History</pluralLabel>
</CustomObject>PermissionSets
No differences here. Existing permissionsets can be used as well to provide access to the Big Objects Data.
<?xml version="1.0" encoding="UTF-8"?>
<PermissionSet xmlns="http://soap.sforce.com/2006/04/metadata">
<fieldPermissions>
<editable>true</editable>
<field>Rider_History__b.Start_Location_Lat__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>Rider_History__b.Start_Location_Long__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>Rider_History__b.End_Time__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>Rider_History__b.Service_Type__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>Rider_History__b.Rider_Rating__c</field>
<readable>true</readable>
</fieldPermissions>
<label>Rider History Permission Set</label>
</PermissionSet>
Package File
Big Objects cannot be specified individually or by API name. I don't think this is as significant a limitation. Since separate packages can be created for Big Objects.
<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
<types>
<members>*</members>
<name>CustomObject</name>
</types>
<types>
<members>Rider_History</members>
<name>PermissionSet</name>
</types>
<version>41.0</version>
</Package>
Deployment of big objects can be done using ant migration tool, Force.com IDE or workbench. Functionality for deployment using changesets might be there when you are reading this blog. At the time of writing changesets do not support deployment of big objects.
Once the Big Object is deployed it can be viewed from the Setup Menu under Big Objects.
Clicking on a big object from the list reveals the objects details including the fields that make up the index, their order in the index, and their sort direction. From this screen edits can be made to the Big Object (name, labels) and the fields. Adding a field or any other significant changes requires going through the defining and deploying process. Big Objects can be deleted from this screen as well. Once deleted the Big Object's data is stored for 15 days and deleted permanently after that.
Populating a Big Object
Big Objects can be populated through a .csv file and the Bulk or SOAP API. Similar to loading non big objects the header (first row) contains the field labels or field api names. Sample file for the Rider_History__b object.
Rider_Account__c,Rider_Rating__c,Service_Type__c,Start_Time__c,End_Time__c
"Test Account 1",3.5,Limousine,2017-11-01T12:01:01Z,2017-11-01T13:01:01Z
"Test Account 2",4.0,SUV,2017-11-01T12:01:01Z,2017-11-01T13:01:01Z
"Test Account 3",4.5,Sedan,2017-11-01T12:01:01Z,2017-11-01T13:01:01Z
Another way of populating big objects is through Apex. The two main functions that are used to accomplish DML for Big Objects are Database.insertImmediate and Database.deleteImmediate. The insertImmediate method provides upsert functionality using the index fields. Reinserting a record with the same index but with different data results in an update. If a record with the index exists, the indexes are overwritten with the index values with the new data.
//Define the record
<Rider_History__b> rh = new Rider_History__b();
rh.Rider_Account__c = "001R000000302D3";
rh.Rider_Rating__c = 2.5;
rh.Service_Type__c = "Taxi";
rh.Start_Time__c = "2017-11-01T23:01:01Z";
rh.End_Time__c = "2017-11-01T00:01:01Z";
//Insert the record, which creates a new record
database.insertImmediate(rh);
//Modify a field in the index
rh.Rider_Account__c = "001R000000542D4";
//Insert the record, creating a new record because the primary key has changed
database.insertImmediate(rh);
//Modify a field not included in the index
rh.Service_Type__c = "Rickshaw";
//Insert the record, which updates the second record because the index is the same
database.insertImmediate(rh);
Querying a Big Object
Big Object data can be queried using SOQL and Async SOQL. Async SOQL was designed specifically for handling large volumes and cannot time out because of their async nature. Async SOQL can query entity data, standard objects, custom objects, and big objects and runs in the background. Implemented via the chatter API.
Usecase | SOQL | Async SOQL |
Visualforce or non-waiting UI changes. | ✓ | |
Need results for use immediately within apex code. | ✓ | |
The query is for a small amount of data. | ✓ | |
Querying against large data volumes. | ✓ | |
Ensure completion of the query. | ✓ | |
No aggregation or filtering outside the index is required | ✓ |
There are some considerations when querying big objects using SOQL:
Only subset of SOQL commands using the big object’s index fields as filters are allowed.
Not all indexed fields are required in the filter criteria, but the ordering of the filters should follow the order of indexed fields in the big object definition.
ID field in an SObject query returns an empty id (000000000000000 or 000000000000000AAA).
Please refer to this link to get more specific details on querying big objects with SOQL.
Async SOQL can be used the same way as any other queries using the chatter api from an external application. Please refer to the link for further details on Async SOQL.
Comments