【论文查重】pycorrector 搭建错别字识别

先更新 wheel ,否则会报下面这个错

error: can’t copy ‘pycorrector/data/kenlm’: doesn’t exist or not a regular file

pip install –upgrade wheel
pip install –upgrade setuptools

pip install pypinyin
pip install kenlm
pip install numpy
pip install jieba
pip install pycorrector

kenlm  在centos下安装可能会出现找不到 Python.h 的错误

解决办法    yum install python36u-devel

python 文件内容

#coding=utf-8
import pycorrector
corrected_sent,detail=pycorrector.correct(‘少先队员因该为老人让坐’)
print(corrected_sent, detail)

执行结果

2019-03-08 10:10:32,814 – /usr/local/lib/python3.5/dist-packages/pycorrector/corrector.py – DEBUG – Loaded same pinyin file: /usr/local/lib/python3.5/dist-packages/pycorrector/data/same_pinyin.txt, same stroke file: /usr/local/lib/python3.5/dist-packages/pycorrector/data/same_stroke.txt, spend: 0.047 s.
2019-03-08 10:10:32,816 – /usr/local/lib/python3.5/dist-packages/pycorrector/detector.py – DEBUG – Loaded language model: /usr/local/lib/python3.5/dist-packages/pycorrector/data/kenlm/people_chars_lm.klm, spend: 0.0011556148529052734 s
2019-03-08 10:10:34,098 – /usr/local/lib/python3.5/dist-packages/pycorrector/detector.py – DEBUG – Loaded word freq file: /usr/local/lib/python3.5/dist-packages/pycorrector/data/word_dict.txt, spend: 1.2822625637054443 s
2019-03-08 10:10:34,099 – /usr/local/lib/python3.5/dist-packages/pycorrector/detector.py – DEBUG – Loaded confusion file: /usr/local/lib/python3.5/dist-packages/pycorrector/data/custom_confusion.txt, spend: 1.2832543849945068 s
少先队员应该为老人让座 [[‘因该’, ‘应该’, 4, 6], [‘坐’, ‘座’, 10, 11]]

机器找不到 libcudnn.so.6

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcudnn.so.6: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 72, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcudnn.so.6: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

.bashrc 中加入

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

pix2code GUI转换为 iOS,android,web 页面源码

GITHUB:  https://github.com/tonybeltramelli/pix2code

操作流程见github,非常详细。

一路无话,在公司用 1070训,报

python3 ./train.py ../datasets/web/training_set ../bin
Using TensorFlow backend.
Loading data...
Generating sparse vectors...
Dataset size: 143741
Vocabulary size: 19
Input shape: (256, 256, 3)
Output size: 19
Convert arrays...
Traceback (most recent call last):
 File "./train.py", line 66, in <module>
 run(input_path, output_path, is_memory_intensive=use_generator, pretrained_model=pretrained_weigths)
 File "./train.py", line 24, in run
 dataset.convert_arrays()
 File "/root/code/pix2code/model/classes/dataset/Dataset.py", line 82, in convert_arrays
 self.input_images = np.array(self.input_images)
MemoryError

nvidia-smi 了下,发现有个进程占了2G显存。kill 之,重试还报一样的错误。

重新看了下 github 上的 readme

解决方案,用下面这个语句。也是官方推荐的train方式,之前没仔细看。

python3 ./train.py ../datasets/web/training_set ../bin 1

开始非常愉快的训练了。。。1080ti 大概 18分钟1轮,默认设定为 10轮。

一夜无话。。。。到早上终于训练好了

截了一张博客的图

生成 GUI

python3 ./sample.py ../bin pix2code  1.jpg  ../code

生成的GUI,似乎生成的是它演示的哪个页面。不是我提供的图。。奇怪。重新训一下试试。

header{
btn-inactive,btn-active
}
row{
single{
small-title,text,btn-green
}
}
row{
quadruple{
small-title,text,btn-orange
}
quadruple{
small-title,text,btn-orange
}
quadruple{
small-title,text,btn-orange
}
quadruple{
small-title,text,btn-orange
}
}
row{
double{
small-title,text,btn-orange
}
double{
small-title,text,btn-orange
}
}

忘了加  python3 ,默认用 python2 训完了报错

Traceback (most recent call last):
  File "./train.py", line 66, in <module>
    run(input_path, output_path, is_memory_intensive=use_generator, pretrained_m
odel=pretrained_weigths)
  File "./train.py", line 51, in run
    model.fit_generator(generator, steps_per_epoch=steps_per_epoch)
  File "/home/endler/code/pix2code/model/classes/model/pix2code.py", line 70, in
 fit_generator
    self.save()
  File "/home/endler/code/pix2code/model/classes/model/AModel.py", line 18, in s
ave
    self.model.save_weights("{}/{}.h5".format(self.output_path, self.name))
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2
580, in save_weights
    raise ImportError('`save_weights` requires h5py.')
ImportError: `save_weights` requires h5py.

哭死,python2.7 没装  h5py。。。还得重训一次。。。。

gogogo,我就不信了。。。

又报这个,不过 权重文件已经生成了。

Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7f3f6f67ae80>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 696, in __del__
TypeError: 'NoneType' object is not callable

反复试验了几次,貌似它只能识别 demo 哪几张图。ㄟ( ▔, ▔ )ㄏ

Age Gender Estimate TF 一个tensorflow 识别年龄的demo

GITHUB:https://github.com//BoyuanJiang/Age-Gender-Estimate-TF

clone 代码。

下载数据集 到  ~/data

wget   https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/static/imdb_crop.tar

加压缩

tar -xvf imdb_crop.tar

数据集预处理

python convert_to_records_multiCPU.py --imdb --nworks 8

训练的时候发现输出的文件为空。

try 了一下

example = tf.train.Example(features=tf.train.Features(feature={
 # 'height': _int64_feature(rows),
 # 'width': _int64_feature(cols),
 # 'depth': _int64_feature(depth),
 'age': _int64_feature(int(ages[index])),
 'gender': _int64_feature(int(genders[index])),
 'image_raw': _bytes_feature(image_raw),
 'file_name': _bytes_feature(str(file_name[index][0]))}))

报错

’40/nm1102140_rm3713850624_1974-7-29_2013.jpg’ has type <class ‘str’>, but expected one of: ((<class ‘bytes’>,),)

去掉

'file_name': _bytes_feature(str(file_name[index][0]))

程序可以正常跑了
训练的时候报错

Invalid argument: Name: <unknown>, Feature: file_name (data type: st
ring) is required but could not be found.

修改 file_name 行如下

'file_name': _bytes_feature(bytes(file_name[index][0],'utf-8'))}))

重新生成。。。

或者换成

python convert_to_records.py –imdb

就好了。

下载models:https://pan.baidu.com/s/1dFewgqH

训练好的model

https://pan.baidu.com/s/1bpllJg7

Train

python3  train.py --lr 1e-3 --weight_decay 1e-5 --epoch 6 --batch_size 128 --keep_prob 0.8 --cuda

Test 选出最好的模型

python3 test.py --images "./data/test" --model_path "./models" --batch_size 128 --choose_best --cuda
Age_MAE:7.07,Gender_Acc:80.92%,Age_model:./models/model.ckpt-12001,Gender_model:./models/model.ckpt-9001

用自己的图片测试

python3 eval.py --I "./demo/demo.jpg" --M "./models/" --font_scale 1 --thickness 1

 

另一个Keras 的实现: Keras 实现的性别年龄检测

KittiSeg Road Segmentation model implemented in tensorflow.

GITHUB:https://github.com/MarvinTeichmann/KittiSeg

git clone https://github.com/MarvinTeichmann/KittiSeg.gi
cd KittiSeg
git submodule update --init --recursive

python download_data.py --kitti_url  http://kitti.is.tue.mpg.de/kitti/data_road.zip

python3 demo.py --input_image data/demo/demo.png

自己找了几张图测试,效果还行。

 

待解决:

  1. 对非公路如石子路识别不理想。
  2. 公路车多的时候,检测不出公路
  3. 识别的时候,似乎没有使用GPU。
    1. 原因是我机器上装了 tensorflow cpu 和gpu 版,gpu 是 1.2  cpu 是1.4 这个项目在1.2上跑不起来。都卸了重装最新版的 tensorflow-gpu。
    2. 安装的时候报错。
    3.  libcudnn.so.6: cannot open shared object file: No such file or directory
    4. https://developer.nvidia.com/rdp/cudnn-download 下载新的 cudnn 安装。
    5. Navigate to your <cudnnpath> directory containing the cuDNN Tar file.
    6. Unzip the cuDNN package.
      $ tar -xzvf cudnn-9.0-linux-x64-v7.tgz
    7. Copy the following files into the CUDA Toolkit directory.
      $ sudo cp cuda/include/cudnn.h /usr/local/cuda/include
      $ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
      $ sudo chmod a+r /usr/local/cuda/include/cudnn.h
      /usr/local/cuda/lib64/libcudnn*
      
      
    8.  sudo ldconfig /usr/local/cuda/lib64  #没有这步,还会出错。
    9. 2017-12-15 12:25:14.185527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
      
      name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
      
      pciBusID: 0000:01:00.0
      
      totalMemory: 10.90GiB freeMemory: 10.46GiB
      
      2017-12-15 12:25:14.185550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
    10. 问题解决了

Self-Driving-Car-Demo 第一个无人驾驶Demo

5分钟视频系列,一个最简单的无人驾驶 demo。

GITHUB:https://github.com/llSourcell/Self-Driving-Car-Demo

环境准备

sudo apt-get install mercurial
hg clone https://bitbucket.org/pygame/pygame
cd pygame
sudo apt-get install python3-dev python3-numpy libsdl-dev libsdl-image1.2-dev \
  libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsmpeg-dev libportmidi-dev \
  libavformat-dev libswscale-dev libjpeg-dev libfreetype6-dev
python3 setup.py build
sudo python3 setup.py install
wget https://github.com/viblo/pymunk/archive/pymunk-4.0.0.zip
pip install pymunk-4.0.0.zip
pip3 install keras
pip3 install git+git://github.com/Theano/Theano.git --upgrade --no-deps
pip3 install h5py

到  pymunk 目录  将 python2 的代码格式转为 python3 的

cd /usr/local/lib/python3.5/dist-packages/pymunk

2to3 -w *.py

回到  Self-Driving-Car-Demo 目录

python3 learning.py
报错
pygame.error: No video mode large enough for 1000x700

修改文件 /flat_game/carmunk.py  根据显卡情况修改。1070 的破卡,所以改的小了点。

width = 500 #1000
height = 350 #700

漫长的训练ing   ( 根据说明大概需要 36 hour)

25000 轮后报错 /(ㄒoㄒ)/~~

OSError: Unable to create file (unable to open file: name = ‘saved-models/164-150-100-50000-25000.h5’, errn
o = 2, error message = ‘No such file or directory’, flags = 13, o_flags = 242)

创建  saved-models 文件夹。重新训练。。。。

pip3 install matplotlib
apt install python3-tk
python3 plotting.py

修改  nn.py 中 model 的名字。

python3 playing.py

 

Tensorflow Android Demo 使用 Yolo

它默认支持三种网络,

TF_OD_API, MULTIBOX, YOLO;

除了 YOLO ,都给了下载链接,想试试 YOLO ,没找到

graph-tiny-yolo-voc.pb

的下载,自己动手丰衣足食。

git clone git@github.com:thtrieu/darkflow.git
cd darkflow
python3 setup.py build_ext --inplace
pip install -e .

下载对应的 yolo weight 文件

wget https://pjreddie.com/media/files/yolo.weights

更多权重文件下载 :https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU

https://pjreddie.com/darknet/yolo/

flow --model cfg/yolo.cfg --load yolo.weights --savepb
Parsing ./cfg/yolo.cfg
Parsing cfg/yolo.cfg
Loading yolo.weights ...
Successfully identified 203934260 bytes
Finished in 0.008934736251831055s
Model has a coco model name, loading coco labels.

Building net ...
Source | Train? | Layer description | Output size
-------+--------+----------------------------------+---------------
 | | input | (?, 608, 608, 3)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 608, 608, 32)
 Load | Yep! | maxp 2x2p0_2 | (?, 304, 304, 32)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 304, 304, 64)
 Load | Yep! | maxp 2x2p0_2 | (?, 152, 152, 64)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 152, 152, 128)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 152, 152, 64)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 152, 152, 128)
 Load | Yep! | maxp 2x2p0_2 | (?, 76, 76, 128)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 76, 76, 256)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 76, 76, 128)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 76, 76, 256)
 Load | Yep! | maxp 2x2p0_2 | (?, 38, 38, 256)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 38, 38, 512)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 38, 38, 256)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 38, 38, 512)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 38, 38, 256)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 38, 38, 512)
 Load | Yep! | maxp 2x2p0_2 | (?, 19, 19, 512)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 19, 19, 512)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 19, 19, 512)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | concat [16] | (?, 38, 38, 512)
 Load | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 38, 38, 64)
 Load | Yep! | local flatten 2x2 | (?, 19, 19, 256)
 Load | Yep! | concat [27, 24] | (?, 19, 19, 1280)
 Load | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 19, 19, 1024)
 Load | Yep! | conv 1x1p0_1 linear | (?, 19, 19, 425)
-------+--------+----------------------------------+---------------
Running entirely on CPU
2017-12-08 17:49:27.012124: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Finished in 5.091068267822266s

Rebuild a constant version ...
Done

训练好的文件在 ./built_graph/ 文件夹

将生成的文件拷贝到

android_asset

编译成功。。。

效果不是很理想,很多东西被认成  person 了。。。

TensorFlow Lite iOS 调试笔记

下载 tensorflow 代码,到目录

/tensorflow/contrib/lite/examples/ios/simple/

 

安装pods

sudo gem install cocoapods
pod install --verbose --no-repo-update

漫长的等待后打开

simple.xcworkspace

报错

tensorflow/contrib/lite/schema/schema_generated.h:7:10: 'flatbuffers/flatbuffers.h' file not found

FlatBuffers 没装

xcode-select --install
brew install automake
brew install libtool

下面命令需要退到 根目录下执行,否则找不到对应目录,另需已安装 wget

tensorflow/contrib/lite/download_dependencies.sh

在 cp 前 莫名 有个 echo ,导致 cp 不执行。需要手动下载这些包,再传到制定目录。

tensorflow/contrib/lite/build_ios_universal_lib.sh

编译不成功,感觉 这个 lite 版的 ios 还没有完工。也可能是我这台 imac的问题,回家用macpro 试试再说。

TensorFlow Demo labels 文件中文版(汉化完成)

使用相关: TensorFlow Lite 初探

试了下各种自动翻译,效果都不理想。还是自己动手丰衣足食。顺带背单词了。

手工翻了一半,用有道翻译了一半。2017.12.7 发现错误会随时更新。

12.8 更新了3个有道翻错的词。
继续阅读TensorFlow Demo labels 文件中文版(汉化完成)