Skip to content
Navigation Menu
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Copilot app
Direct agents from issue to merge
MCP Registry
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
Code Quality
Enforce quality at merge
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
Accelerator
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
Search
/
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading.
Please reload this page
.
mideind
/
Tokenizer
Public
Notifications
You must be signed in to change notification settings
Fork
9
Star
32
Code
Issues
0
Pull requests
0
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights
Actions: mideind/Tokenizer
Actions
All workflows
Workflows
Copilot code review
Copilot code review
Dependency Graph
Dependency Graph
release
release
tests
tests
Show more workflows...
Management
Caches
Deployments
tests
tests
Actions
Loading...
Loading
Sorry, something went wrong.
Uh oh!
There was an error while loading.
Please reload this page
.
will be ignored since log searching is not yet available
Show workflow options
Create status badge
Create status badge
Loading
Uh oh!
There was an error while loading.
Please reload this page
.
python-package.yml
will be ignored since log searching is not yet available
70 workflow runs
70 workflow runs
Event
Filter by Event
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching events.
Status
Filter by Status
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching statuses.
Branch
Filter by Branch
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching branches.
Actor
Filter by Actor
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching users.
Merge pull request #62 from mideind/perf/profile-tokenizer-hotspots
tests
#271:
Commit
7ce5d94
pushed by
vthorsteinsson
25s
master
master
25s
View workflow file
Release Tokenizer 3.6.4
tests
#270:
Pull request
#62
opened by
vthorsteinsson
24s
perf/profile-tokenizer-hotspots
perf/profile-tokenizer-hotspots
24s
View #62
View workflow file
Prepare Tokenizer 3.6.3 release
tests
#269:
Commit
b374eb1
pushed by
vthorsteinsson
33s
master
master
33s
View workflow file
Prepare Tokenizer 3.6.3 release
tests
#268:
Commit
b374eb1
pushed by
vthorsteinsson
43s
fix-modernize-tokenizer
fix-modernize-tokenizer
43s
View workflow file
Version 3.6.2: Add py.typed marker for PEP 561
tests
#267:
Commit
030dcbb
pushed by
vthorsteinsson
24s
master
master
24s
View workflow file
CI: Add PyPI release workflow with trusted publishing
tests
#266:
Commit
fe6870b
pushed by
vthorsteinsson
26s
master
master
26s
View workflow file
Version 3.6.1: Add py.typed marker for PEP 561
tests
#265:
Commit
982f50b
pushed by
vthorsteinsson
27s
master
master
27s
View workflow file
Version 3.6.0: Remove deprecated handle_kludgy_ordinals option
tests
#264:
Commit
7bffb58
pushed by
vthorsteinsson
28s
master
master
28s
View workflow file
Merge pull request #61 from mideind/remove-kludgy
tests
#263:
Commit
09bf1e5
pushed by
vthorsteinsson
26s
master
master
26s
View workflow file
Remove deprecated --handle_kludgy_ordinals flag
tests
#262:
Pull request
#61
synchronize by
vthorsteinsson
24s
remove-kludgy
remove-kludgy
24s
View #61
View workflow file
Simplify KLUDGY_ORDINALS to a tuple for use with startswith()
tests
#261:
Commit
ceabab6
pushed by
vthorsteinsson
22s
remove-kludgy
remove-kludgy
22s
View #61
View workflow file
Remove deprecated --handle_kludgy_ordinals flag
tests
#260:
Pull request
#61
synchronize by
vthorsteinsson
29s
remove-kludgy
remove-kludgy
29s
View #61
View workflow file
CI: Fix PyPy 3.11 build by avoiding mypy dependency
tests
#259:
Commit
39a39e5
pushed by
vthorsteinsson
24s
remove-kludgy
remove-kludgy
24s
View workflow file
Remove deprecated --handle_kludgy_ordinals flag
tests
#258:
Pull request
#61
opened by
vthorsteinsson
25s
remove-kludgy
remove-kludgy
25s
View #61
View workflow file
Remove deprecated --handle_kludgy_ordinals flag
tests
#257:
Commit
f9bb989
pushed by
vthorsteinsson
22s
remove-kludgy
remove-kludgy
22s
View workflow file
Version 3.5.5: Maintenance release
tests
#256:
Commit
28ce86e
pushed by
vthorsteinsson
46s
master
master
46s
View workflow file
Removed obsolete 'type: ignore' comments
tests
#255:
Commit
1524b7e
pushed by
vthorsteinsson
46s
master
master
46s
View workflow file
CI: Remove PyPy 3.9 and 3.10 from build matrix
tests
#254:
Commit
da2197a
pushed by
vthorsteinsson
59s
master
master
59s
View workflow file
Fixed mypy complaints
tests
#253:
Commit
184c999
pushed by
vthorsteinsson
48s
master
master
48s
View workflow file
Updated and modernized pyproject.toml
tests
#252:
Commit
ec196ac
pushed by
vthorsteinsson
30s
master
master
30s
View workflow file
Version 3.5.4: Improved dash and hyphen handling
tests
#251:
Commit
5fcb625
pushed by
vthorsteinsson
27s
master
master
27s
View workflow file
Merge pull request #60 from mideind/fix-hyphens
tests
#250:
Commit
2ded9f1
pushed by
vthorsteinsson
27s
master
master
27s
View workflow file
Version 3.5.4: Improved dash and hyphen handling
tests
#249:
Pull request
#60
synchronize by
vthorsteinsson
27s
fix-hyphens
fix-hyphens
27s
View #60
View workflow file
Minor review correction
tests
#248:
Commit
9de779f
pushed by
vthorsteinsson
27s
fix-hyphens
fix-hyphens
27s
View #60
View workflow file
Version 3.5.4: Improved dash and hyphen handling
tests
#247:
Pull request
#60
synchronize by
vthorsteinsson
28s
fix-hyphens
fix-hyphens
28s
View #60
View workflow file
Previous
1
2
3
Next
You can’t perform that action at this time.