React Native plugin for adding voice using [Spokestack](https://www.spokestack.io). This includes speech recognition, wakeword, and natural language understanding, as well as synthesizing text to speech using Spokestack voices.
## Requirements
- _React Native_: 0.60.0+
- _Android_: Android SDK 21+
- _iOS_: iOS 13+
## Installation
[](https://www.npmjs.com/package/react-native-spokestack)
Using npm:
```sh
npm install --save react-native-spokestack
```
or using yarn:
```sh
yarn add react-native-spokestack
```
Then follow the instructions for each platform to link react-native-spokestack to your project:
## iOS installation
iOS details
### Set deployment target
First, open XCode and go to Project -> Info to set the iOS Deployment target to 13.0 or higher.
Also, set deployment to 13.0 under Target -> General -> Deployment Info.
### Remove invalid library search path
When Flipper was introduced to React Native, some library search paths were set for Swift. There has been a longstanding issue with the default search paths in React Native projects because a search path was added for swift 5.0 which prevented any other React Native libraries from using APIs only available in Swift 5.2 or later. Spokestack-iOS, a dependency of react-native-spokestack makes use of these APIs and XCode will fail to build.
Fortunately, the fix is fairly simple. Go to your target -> Build Settings and search for "Library Search Paths".
Remove `"\"$(TOOLCHAIN_DIR)/usr/lib/swift-5.0/$(PLATFORM_NAME)\""` from the list.
### Edit Podfile
Before running `pod install`, make sure to make the following edits.
react-native-spokestack makes use of relatively new APIs only available in iOS 13+. Set the deployment target to iOS 13 at the top of your Podfile:
```ruby
platform :ios, '13.0'
```
We also need to use `use_frameworks!` in our Podfile in order to support dependencies written in Swift.
```ruby
target 'SpokestackExample' do
use_frameworks!
#...
```
For now, `use_frameworks!` does not work with Flipper, so we also need to disable Flipper. Remove any Flipper-related lines in your Podfile. In React Native 0.63.2+, they look like this:
```ruby
# X Remove or comment out these lines X
# use_flipper!
# post_install do |installer|
# flipper_post_install(installer)
# end
# XX
```
#### Bug in React Native 0.64.0 (should be fixed in 0.64.1)
React Native 0.64.0 broke any projects using `use_frameworks!` in their Podfiles.
For more info on this bug, see https://github.com/facebook/react-native/issues/31149.
To workaround this issue, add the following to your Podfile:
```ruby
# Moves 'Generate Specs' build_phase to be first for FBReactNativeSpec
post_install do |installer|
installer.pods_project.targets.each do |target|
if (target.name&.eql?('FBReactNativeSpec'))
target.build_phases.each do |build_phase|
if (build_phase.respond_to?(:name) && build_phase.name.eql?('[CP-User] Generate Specs'))
target.build_phases.move(build_phase, 0)
end
end
end
end
end
```
#### pod install
Remove your existing Podfile.lock and Pods folder to ensure no conflicts, then install the pods:
```sh
$ npx pod-install
```
### Edit Info.plist
Add the following to your Info.plist to enable permissions.
```xml
NSMicrophoneUsageDescriptionThis app uses the microphone to hear voice commandsNSSpeechRecognitionUsageDescriptionThis app uses speech recognition to process voice commands
```
#### Remove Flipper
While Flipper works on fixing their pod for `use_frameworks!`, we must disable Flipper. We already removed the Flipper dependencies from Pods above, but there remains some code in the AppDelegate.m that imports Flipper. There are two ways to fix this.
1. You can disable Flipper imports without removing any code from the AppDelegate. To do this, open your xcworkspace file in XCode. Go to your target, then Build Settings, search for "C Flags", remove `-DFB_SONARKIT_ENABLED=1` from flags.
1. Remove all Flipper-related code from your AppDelegate.m.
In our example app, we've done option 1 and left in the Flipper code in case they get it working in the future and we can add it back.
### Edit AppDelegate.m
#### Add AVFoundation to imports
```objc
#import
```
#### AudioSession category
Set the AudioSession category. There are several configurations that work.
The following is a suggestion that should fit most use cases:
```objc
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
AVAudioSession *session = [AVAudioSession sharedInstance];
[session setCategory:AVAudioSessionCategoryPlayAndRecord
mode:AVAudioSessionModeDefault
options:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowAirPlay | AVAudioSessionCategoryOptionAllowBluetoothA2DP | AVAudioSessionCategoryOptionAllowBluetooth
error:nil];
[session setActive:YES error:nil];
// ...
```
## Android installation
Android details
### ASR Support
The example usage uses the system-provided ASRs (`AndroidSpeechRecognizer` and `AppleSpeechRecognizer`). However, `AndroidSpeechRecognizer` is not available on 100% of devices. If your app supports a device that doesn't have built-in speech recognition, use Spokestack ASR instead by setting the `profile` to a Spokestack profile using the `profile` prop.
See our [ASR documentation](https://www.spokestack.io/docs/concepts/asr) for more information.
### Edit root build.gradle (_not_ app/build.gradle)
```groovy
// ...
ext {
// Set the minimum SDK to 24.
// React Native 0.64+ sets version 21.
// If you prefer to leave the minimum SDK at 21,
// another option is to set this to 21, but
// also set android.enableDexingArtifactTransform=false
// in your top-level gradle.properties.
// See https://github.com/google/ExoPlayer/issues/6801#issuecomment-574089568
minSdkVersion = 24
// ...
dependencies {
// Minimium gradle is 3.0.1+
// The latest React Native already has this
classpath("com.android.tools.build:gradle:4.2.1")
```
### Edit AndroidManifest.xml
Add the necessary permissions to your `AndroidManifest.xml`. The first permission is often there already. The second is needed for using the microphone.
```xml
```
### Request RECORD_AUDIO permission
The RECORD_AUDIO permission is special in that it must be both listed in the `AndroidManifest.xml` as well as requested at runtime. There are a couple ways to handle this (react-native-spokestack does not do this for you):
1. **Recommended** Add a screen to your onboarding that explains the need for the permissions used on each platform (RECORD_AUDIO on Android and Speech Recognition on iOS). Have a look at [react-native-permissions](https://github.com/zoontek/react-native-permissions) to handle permissions in a more robust way.
2. Request the permissions only when needed, such as when a user taps on a "listen" button. Avoid asking for permission with no context or without explaining why it is needed. In other words, we do not recommend asking for permission on app launch.
While iOS will bring up permissions dialogs automatically for any permissions needed, you must do this manually in Android.
React Native already provides a module for this. See [React Native's PermissionsAndroid](https://reactnative.dev/docs/permissionsandroid) for more info.
## Usage
[Get started using Spokestack](https://www.spokestack.io/docs/React%20Native/getting-started), or check out our in-depth tutorials on [ASR](https://www.spokestack.io/docs/React%20Native/speech-pipeline), [NLU](https://www.spokestack.io/docs/React%20Native/nlu), and [TTS](https://www.spokestack.io/docs/React%20Native/tts). Also be sure to take a look at the [Cookbook](https://www.spokestack.io/docs/React%20Native/cookbook) for quick solutions to common problems.
A working example app is included in this repo in the `example/` folder.
```js
import Spokestack from 'react-native-spokestack'
import { View, Button, Text } from 'react-native'
function App() {
const [listening, setListening] = useState(false)
const onActivate = () => setListening(true)
const onDeactivate = () => setListening(false)
const onRecognize = ({ transcript }) => console.log(transcript)
useEffect(() => {
Spokestack.addEventListener('activate', onActivate)
Spokestack.addEventListener('deactivate', onDeactivate)
Spokestack.addEventListener('recognize', onRecognize)
Spokestack.initialize(
process.env.SPOKESTACK_CLIENT_ID,
process.env.SPOKESTACK_CLIENT_SECRET
)
// This example starts the Spokestack pipeline immediately,
// but it could be delayed until after onboarding or other
// conditions have been met.
.then(Spokestack.start)
return () => {
Spokestack.removeAllListeners()
}
}, [])
return (
)
}
```
## Including model files in your app bundle
To include model files locally in your app (rather than downloading them from a CDN), you also need to add the necessary extensions so
the files can be included by Babel. To do this, edit your [`metro.config.js`](https://facebook.github.io/metro/docs/configuration/).
```js
const defaults = require('metro-config/src/defaults/defaults')
module.exports = {
resolver: {
assetExts: defaults.assetExts.concat(['tflite', 'txt', 'sjson'])
}
}
```
Then include model files using source objects:
```js
Spokestack.initialize(clientId, clientSecret, {
wakeword: {
filter: require('./filter.tflite'),
detect: require('./detect.tflite'),
encode: require('./encode.tflite')
},
nlu: {
model: require('./nlu.tflite'),
vocab: require('./vocab.txt'),
// Be sure not to use "json" here.
// We use a different extension (.sjson) so that the file is not
// immediately parsed as json and instead
// passes a require source object to Spokestack.
// The special extension is only necessary for local files.
metadata: require('./metadata.sjson')
}
})
```
This is not required. Pass remote URLs to the same config options and the files will be downloaded and cached when first calling `initialize`.
## Contributing
See the [contributing guide](CONTRIBUTING.md) to learn how to contribute to the repository and the development workflow.
---
# API Documentation
### initialize
▸ **initialize**(`clientId`, `clientSecret`, `config?`): `Promise`<`void`\>
Initialize the speech pipeline; required for all other methods.
The first 2 args are your Spokestack credentials
available for free from https://spokestack.io.
Avoid hardcoding these in your app.
There are several ways to include
environment variables in your code.
Using process.env:
https://babeljs.io/docs/en/babel-plugin-transform-inline-environment-variables/
Using a local .env file ignored by git:
https://github.com/goatandsheep/react-native-dotenv
https://github.com/luggit/react-native-config
See [SpokestackConfig](#SpokestackConfig) for all available options.
**`example`**
```js
import Spokestack from 'react-native-spokestack'
// ...
await Spokestack.initialize(process.env.CLIENT_ID, process.env.CLIENT_SECRET, {
pipeline: {
profile: Spokestack.PipelineProfile.PTT_NATIVE_ASR
}
})
```
#### Parameters
| Name | Type |
| :------------- | :-------------------------------------- |
| `clientId` | `string` |
| `clientSecret` | `string` |
| `config?` | `[SpokestackConfig](#SpokestackConfig)` |
#### Returns
`Promise`<`void`\>
#### Defined in
[index.ts:64](https://github.com/spokestack/react-native-spokestack/blob/b127e05/src/index.ts#L64)
### destroy
▸ **destroy**(): `Promise`<`void`\>
Destroys the speech pipeline, removes all listeners, and frees up all resources.
This can be called before re-initializing the pipeline.
A good place to call this is in `componentWillUnmount`.
**`example`**
```js
componentWillUnmount() {
Spokestack.destroy()
}
```
#### Returns
`Promise`<`void`\>
#### Defined in
[index.ts:81](https://github.com/spokestack/react-native-spokestack/blob/b127e05/src/index.ts#L81)
### start
▸ **start**(): `Promise`<`void`\>
Start the speech pipeline.
The speech pipeline starts in the `deactivate` state.
**`example`**
```js
import Spokestack from 'react-native-spokestack`
// ...
Spokestack.initialize(process.env.CLIENT_ID, process.env.CLIENT_SECRET)
.then(Spokestack.start)
```
#### Returns
`Promise`<`void`\>
#### Defined in
[index.ts:96](https://github.com/spokestack/react-native-spokestack/blob/b127e05/src/index.ts#L96)
### stop
▸ **stop**(): `Promise`<`void`\>
Stop the speech pipeline.
This effectively stops ASR, VAD, and wakeword.
**`example`**
```js
import Spokestack from 'react-native-spokestack`
// ...
await Spokestack.stop()
```
#### Returns
`Promise`<`void`\>
#### Defined in
[index.ts:110](https://github.com/spokestack/react-native-spokestack/blob/b127e05/src/index.ts#L110)
### activate
▸ **activate**(): `Promise`<`void`\>
Manually activate the speech pipeline.
This is necessary when using a PTT profile.
VAD profiles can also activate ASR without the need
to call this method.
**`example`**
```js
import Spokestack from 'react-native-spokestack`
// ...